Minutes UNICORE integration task force 2011-06-15 15:00-16:30 Minutes belonging to Agenda under https://www.egi.eu/indico/conferenceDisplay.py?confId=498 Presence: Michaela Barth MB Andrew Lukoshko AL Rebecca Breu RB Gert Svensson GS Krzysztof Benedyczak KB Emir Imamagic EI John Gordon JG Excused: (Foued Jrad FR) David Meredith DM MB: Going through last meetings minutes: 1) Last meetings minutes https://www.egi.eu/indico/getFile.py/access?resId=0&materialId=minutes&confId=465 2) Going through last meetings Action points https://wiki.egi.eu/wiki/UNICORE_integration_task_force MB: AP: "EMI registry use case", last Update: EMI ESS reviewed, currently no PGI-wg meetings. PGI working group very quite right now, no date for a next meeting set. KB: EMI registry won't contain any static info, this info is contained in the services registered in the local and global registries. It is not static in the sense that it is entered in the GOCDB, that is the info I got from Laurence. MB: Yes, right. MB: AP: "ServiceEndpointURLs in GOCDB", last Update: Update: Requirement has been fixed and seems to work: ttps://goc.gridops.org/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0 Has Poland alread tried to add some services using ServiceEndpointURLs as well? KB: In Poland we didn't add them to GOCDB yet, as EGI is trying to force NGIs to clean up their GOCDB contents. We just checked that the URL is there. We decided to wait to get official info from EGI about certification of sites with UNICORE services, how availability is configured and about accounting and Nagios monitoring. It looks like this feature is fully okay from implementation site, though. MB: There is no explicit information for UNICORE sites, the same site certification procedure as for everybody else is valid. It was tried to make this last version of the site certification procedure as middleware independent as possible. KB: The details are still unclear: ACE computing (called gridview in the past) for example uses some services like CE and SE. How are UNICORE services to be grouped there? EI: From May on the availability is calculated in ACE. KB: I also had the feeling that this ACE is a really important tool. The availability is computed using ACE, so I think it is quite important for us. EI: These are just details. ACE will take care of UNICORE CE as soon as the probes are integrated. You won't have availability in the first two months, because this is according to current EGI procedures. KB: This is exactly what we want. We want to wait for these tools to be available. Of course, we have one uncertified site where we can add any services for testing. There was an email from EGI about fake/unused sites and we had to say that it is needed. EI. The test site is probably needed by me to test the probes during integration, as well. MB: This won't be a problem within the next two months. KB: Agreed. MB: It would be good to have the services added to this test site. KB: I can forward your requests to the admins. MB: So, we possibly need a new AP here, and should close this one. MB: * AP: "Regular probe integration ticket updates" for everybody (especially EI, KB and FR): to keep https://rt.egi.eu/rt/Ticket/Display.html?id=306 updated. KB: I can give an oral status update: The probes for SAM Nagios are going to be maintained by EMI, this was decided inside EMI. It was also decided who will do this for UNICORE: it will be done by the person currently doing this work within PL-GIRD. This person will be transferred to EMI. MB: Thank you. KB: one more thing: Morris wrote in an email that there are also UNICORE Nagios probes developed in Karlsruhe. Rebecca can you add something? RB: Those were those developed by Foued. I don't know what the status is there. MB-> new AP: to ask Foued about status of probes in Karlsruhe, if there are there some further probes that need to go over to EMI. KB: I think the AP can be closed: the AP is now being carried in EMI, the process to pass probes will be moved to EMI directly. MB: Then I would like to make an AP on one of you two to update the ticket and try to close it. EI: Once we do a release, then I'll close it. If you Krzysztof, could just update it? KB: I can only say something about the UNICORE probes. EI: We don't care about the other ones. KB: For UNICORE it was just decided a few days ago. EI: I'll close it once we do a release. MB: * AP: EI to contact Maciej Pawlik and PaweÅ‚ Wolniewicz to discuss sensible Nagios configuration default definitions. EI: That one just didn't happen. I haven't managed to do it. I guess you can close it, since it is part of the integration point I will need tho contact eventually. This AP was mainly needed when we thought GOCDB was taking longer. MB: Ok, I'll close it. MB: * AP: "Keep an eye on accounting": See that EMI follows its roadmap and we get the possibility to send accounting different from gLite to APEL. Related to get updated on the work going on in the new EMI Workinggroup led by Cristina https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccounting KB: AFAIK, nothing happened recently, by the end of next months it should because the deadlines for the unified records format is end of this month and end of next month. MB: But you made this list with what is needed to compare between the different middlewares right? KB: Yes, yes, nothing happened since then. MB: * AP: Start a more tight collaboration with Belarus. AL to keep us updated on efforts to go open source or other possibilities to collaborate, KB to send info on new release with all major features. MB: I've seen Krzysztof sent this info to the mailinglist. Andrew how is it going with the open source efforts? AL: Currently the system isn't open source. I will have some problems, our head of institution disallowed me to do this. After a week I was disallowed, because there are still some legal issues with sharing the source. Our direction is not very satisfied with cooperation with EGI, maybe the decision not to share source is based on personal opinions. I have written an email to Morris Riedel asking for help. Some papers should be prepared with authority and property stuff. And they should be signed by both EU and Belarus participants of EGI. The big problem I think is the satisfaction with the cooperation. AL: I tried the polish solution for accounting and have written some notes about obvious mistakes in the documentation part. Since then I haven't seen these services and documentation. And they have not reacted yet. KB: We also tested this latest release, and unfortunately I must say I was personally not satisfied with the documentation and some small minor bugs. All of Andrews remarks were also noted and will be included/taken into account in the next major update. Documentation will be four times longer and more detailed, and the update will include fixes to some smaller bugs, and show improved usability in the preferences. It will take some time. This is pending of a longer vacation of the main developer. He will return 7th of July. So I suspect the update to be ready middle of July or around 20th of July. There are a lot of small things that should be carefully tested. We also got some feedback from Jülich of people who tested it there. MB: Thank you Krzysztof that is good, but I am sorry to hear that the Belarus open source efforts are being hampered. Andrew, should we try to contact your director directly? Do you think that would be worthwile? AL: It seems they are not satisfied with the current financial support. MB: I'll write Tiziana or maybe even Steven Newhouse to contact him and find out the real reasons why the decision to go open source was backed. If you could just provide some contact information. AL: I will. Update: AP: AL to provide some contact information to his director MB: AP: "Best practices", I have no current update here and I haven't found time yet to join the ongoing documentation workshop in Zürich. Somebody else here who possibly phoned in? MB: So, nobody had time yet to join the documentation workshop. MB: * AP: "Next Meeting": MB making again a doodlepoll for next meeting with Global Timezones and maybe function enabled. Update: http://www.doodle.com/7xcgzsqxadp45v6q --> can be closed. MB: * AP: "List of service types to be integrated." And there we are actually already in the next point in the agenda 3) Discussion of Progress: a) GOCDB David has updated the relevant RT ticket and has added the required set of unicore6 services to GOCDB given by Krzysztof. See: https://rt.egi.eu/rt/Ticket/Display.html?id=944 https://wiki.egi.eu/wiki/GOCDB/Input_System_User_Documentation#Service_types Remaining TODOs: * a sentence for each service type describing what it is. * Decide if more/other unicore6 services are required for GOCDB. unicore6.UNICOREX unicore6.Gateway unicore6.Registry unicore6.TargetSystemFactory unicore6.StorageFactory unicore6.StorageManagement unicore6.ServiceOrchestrator unicore6.WorkflowFactory unicore6.UVOSAssertionQueryService MB: I posted some suggestions for service type descriptions in the Agenda I would like to discuss with you. First of all the removal of unicore6.UNICOREX. Why should it be now removed? It was a lot of effort to get it in into GOCDB in the first place, why don't we want it anymore? KB: UNICORE X is a name of something you can download and install, it is somehting like a set of services, a container, a conglomerate. Now with the GLUE2 services, UNICORE X is a little bit too undefined. It can e.g. include an SMS, a storage factory and a TSF. It can be a wild mix of those services with an undefined number of those services. If you have an UNICORE X you still don't know nothing about the services that are inside. RB: I agree, I also support the removal of this service type. MB: Ok, next: unicore6.TargetSystemFactory : is listed in the Registry and can add new Target System Services (TSSs) was the description I found. KB: This is a service, I admit the name is not very nice for the end users, but this service is used as an entrypoint for submitting jobs, creating TSSs and submitting jobs to those TSSs. MB: unicore6.StorageFactory : provides filespace for workflow execution. For load and data distribution multiple storage factories should be available. KB: It can be used like this, but the main use is in creating storagem management instances: user can create his or her own private storage service with the possibility to destroy and administer her storage. It is often used for workflow execution. A user can create dynamic storage management services for own purposes. Just remove the second sentence. The storage factory itself is very lightweight. It creates SMSes which are requiring storage resources, though. So after creating many you need a lot of disk space. MB: unicore6.StorageManagement : same as SMS? created by StorageFactory, provides an abstract filesystem-like view on a storage resource KB: Yes, that is the same as SMS. It is not always created by a storage factory, it can be also be configured in a static way with a config file. Especially when there is no storage factory at all. The second part is perfectly true. MB: unicore6.ServiceOrchestrator : handles dispatching of a workflow's subjobs, and brokering. Normally there is one per grid infrastructure. KB: Instead of subjobs you could say atomic jobs. Otherwise fine like it is. MB: unicore6.WorkflowFactory : ? I couldn't find any description at all. KB: A Workflow Factory is an entrypoint to the workflow service. The Workflow factory is creating workflow instances and can submit workflows there. It is the entry point for workflow submissions (like the target system facotry for single job submission). MB: unicore6.UVOSAssertionQueryService : The UNICORE VO Service (UVOS) uses the SAML standard and can be used for authorization KB: UVOS is not used per se for authorization, it is just providing data used for authorization, it is a service providing user information. It can be used to provide information on the local account of the user on the target, what groups should the used and so on. MB: why does it have such a long name, why isn't it just called unicore6.UVOS? KB: It has this long name because of the GLUE2 names are the same names as they are used in the EMI registry. We use the real names of the services. MB: --> AP on me to update the ticket with the revised suggestions for the naming. MB: " * Decide if more/other unicore6 services are required for GOCDB." So are we fine now with this last list or what more is missing. KB: I wrote a long email trying to explain all the services. For now it should be enough, some minor probes are missing, some not that critical services are missing. Possibly 1-3 additional service are needed later. But it is a quite good starting point. MB: We should then also already really enter the current services into GOCDB. RB: We will try to add some more. MB: b) Monitoring EI: EMI will take over the probes. As we've heard the exisitng probes of PL-Grid should be integrated in SAM update 12, which is in process. It should be using GOCDB as the topology information source. KB: Do you have a timeline? EI: Update 12 should be at the beginning of July, (1st of july). MB: c) Staged roll-out I've seen the first staged roll-out requests for UNICORE services were issued. Rebecca how is it going? RB: They were verified, we haven't started with it yet. MB: Is it just you now doing it? RB: Between Poland and Germany, yes, we did all of them. MB: d) Accounting KB: The release with the final architecture was released, and the next update will be released during July. JG: I think I need a discussion with Krzysztof on how to best integrate the UNICORE accounting solution, there is no point in having a stand-alone application for accounting. The EMI Computing accounting working group has gathered the differences in the UR format on the different systems. We are still working out on what is required for anybody to publish accounting. We are trying to document it, and are building a wiki together. We are about to start a discussion group. MB: So, John, you will organize something as soon as you have this? JG: Yes. KB: Most important is the definition of this common UR format. The rest should be quite forward later on. JG: Sorry for being late and leaving now again. MB: No problem. MB: e) Argus KB: A few timelines: It was more or less agreed, that this year after the end of summer ( (month 18, September or October) Argus is going to support all required features by UNICORE and ARC. More or less by this time. UNICORE support for ARGUS is going to be updated from EMI authorization, this is a minor effort: just changing some constants; UNICORE should be able to happily talk with Argus. KB: Those changes are really minor. What was finished in April/May was a fine-grained profile which defines all the standard names and attributes. Up to now there was e.g only one profile for CE. Now all attributes and names are defined. So what is remaining is just changing a few constants. MB: That sounds promising. MB: f) Best Practices MB: I guess nothing new here, or can the new documentation for the UNICORE accounting be of some value here? KB: The documentation of accounting is not relevant for best practices. MB: 4) UNICORE summit http://www.unicore.eu/summit/2011/ at Nicolaus Copernicus University, Torun, Poland, on July 7th - 8th EI: No, we won't go there, we don't have funding AL: Same here. RB: The UNICORE summit is once a year. MB: 5) Next meeting a) fixed at the EGI TF, Lyon, week 38 b) summerbreaks? --> Some discussion were it was decided to make doodlepolls for weeks 28 and 30. MB: 6) AOB KB: I have one question which is not strictly related: I'm curious if it is possible to propose other than EGI service types in GOCDB, like NGI specific service types as we have it in Poland. This would allow us to set up alarms in the dashboard. EI: I think this would need a regional GOCDB. There is at least a couple of NGIs, if you submit a new service typ that only you use, this should be defined in the regional gocdb. I suppose it is either the regionalisation taskforce dealing with these issues. Otherwise it would be best to submit a requirement to the RT or ask on the OTAG mailinglist. If it is not possible, then something will have to be developled on the central instance. KB: We can provide and configure probes ourselves, without them seeing. EI: And myEGI will not show them because it won't accept any service which is not in GOCDB. The same logic is true for the national dashboard. You would have to define a completely differnet set of test of operation in you regional portal. MB: I suggest to put this into an requirement. After all, it is not completely unreasonable. KB: We were just curious. EI: Yes, RT is the way to go. MB: Ok, if there is nothing else, thank you all for this meeting. Next week I will be on vacation, btw. -------------------------------------------------------------- Opened/Updated Actionpoints after this meeting: -------------------------------------------------------------- * AP: Add more UNICORE services into GOCDB. * AP: "Regular probe integration ticket updates" for everybody (especially EI, KB and FR): to keep https://rt.egi.eu/rt/Ticket/Display.html?id=306 updated. : Update: KB to make an update that the probes will be maintained by EMI, EI to close the ticket after the corresponding release * AP: "Keep an eye on accounting": See that EMI follows its roadmap and we get the possibility to send accounting different from gLite to APEL. Related to get updated on the work going on in the new EMI Workinggroup led by Cristina https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccounting : Update: Important deadlines end of June and end of July. A wiki should come out of it and a discussion group. JG will take initiative to organize something when the time has come to discuss the details on how to best integrate the current UNICORE accounting solution. * AP: Start a more tight collaboration with Belarus. AL to keep us updated on efforts to go open source or other possibilities to collaborate, KB to send info on new release with all major features. : Update: A new release of UNICORE RUS accounting system is available from sourceforge. It is major milestone as it completes our move to JMS based architecture with a full fail-over support. It is also updated to USE and 6.4.0 release of UNICORE. http://unicore-dev.zam.kfa-juelich.de/documentation/rus-accounting-1.3.1/ : Update: The NGI_BY UNICORE Accounting service AL presented at the EGI UF 2011 got permission to go Open Source. An English version and translated documentation is being prepared for initial export to Sourceforge. NGI-DE already became a non open source version to have some tests and do some comparison. : Update: Allowance to go open source was drawn back after a week, AP on AL to send contact information do deciding director to closer investigate the reasons behind this decision. * AP: "List of service types to be integrated." MB compiling a list of the next service types which should be integrated into GOCDB and sending it to our mailinglist for discussion : Update: Some discussion of this list during the meeting, AP on KB to send his list to the mailinglist as well and to continue discussing there and to put the output of this discussion in ticket https://rt.egi.eu/rt/Ticket/Display.html?id=944 : Update: remaining service name description discussed during the meeting. MB to update the ticket. Decision that currently no more are needed. * AP: "Next Meeting": MB making a doodlepoll for the weeks 28 and 30 (Global Timezones and maybe function enabled). (* AP: KB to create an RT requirement ticket about integration of NGI specific service types into the regional dashboard.) -------------------------------------------------------------- Closed Actionpoints after this meeting: -------------------------------------------------------------- * AP: "ServiceEndpointURLs in GOCDB" Reopen GOCDB requirement ticket for EndPointServiceURL https://rt.egi.eu/rt/Ticket/Display.html?id=975 and ask for second solution (Add URL field for ServiceEndpoint and repeat ServiceEndpoint to represent different URLs /Associate a ServiceEndpoint with a new „1-to-many‟ EndpointLocation entity), with a proposed timeline of 6 months. : Progress: David provided us with some updated slides on the possible implementation with more detail: https://wiki.egi.eu/wiki/File:GocdbGlue2Unicore.pdf : Update: Personal discussion at the EGI UF, Vilnius with John Casson: A new proposed solution with getting rid of the primary key constraint and a new timeline of one month has been discussed. It fulfills all our requirements. This requirement has now top-priority within GOCDB development. : Update: Requirement has been fixed and seems to work: https://goc.gridops.org/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0 : Update: closed. New AP created to add more UNICORE services into GOCDB. *AP: EI to contact Maciej Pawlik and PaweÅ‚ Wolniewicz to discuss sensible Nagios configuration default definitions. : Update: redundant because of fast GOCDB integration, will eventually be done anyway. Closed. *AP: "Next Meeting": MB making again a doodlepoll for next meeting with Global Timezones and maybe function enabled. : Update: http://www.doodle.com/7xcgzsqxadp45v6q