Minutes UNICORE integration task force 2011-07-14 10:00-11:30 Minutes belonging to Agenda under https://www.egi.eu/indico/conferenceDisplay.py?confId=545 Presence: Michaela Barth MB Rebecca Breu RB David Meredith DM Krzysztof Benedyczak KB Mathilde Romberg MR Peter Solagna PS Andrew Lukoshko AL John Gordon JG Tiziana Ferrari TF Emir Imamagic EI on vacation: Gert Svensson GS MB: Let's start with last meetings minutes 1) Last meetings minutes https://www.egi.eu/indico/materialDisplay.py?materialId=minutes&confId=498 MB: I received some corrections, so I guess they are okay now. Any objections? MB: 2) Going through open Action points https://wiki.egi.eu/wiki/UNICORE_integration_task_force MB: Going to the first open action point, the one about the "EMI registry use case", I asked on the mailinglist if this is now still relevant, but haven't received any answer. Could you please have your comments now, with the ServicePointURL in GOCDB working is this action point actually still important to us? Otherwise I can say that the PGI working group within OGF is planning to take up their work again with regular phoneconferences now after OGF 32. Next week or the week after next week is in discussion. KB: The EMI registry will have to support GLUE 2 anyway, for now it is out of our scope, I think you can close this actionpoint. MB: Next action point: "Adding more UNICORE services into GOCDB." Krzysztof reported a bug with entering all RFC 3986 characters in the Service Point URL field which was fixed by David. There was then still something with the description. David, is that fixed now, too? DM: Problem with the description? I am not aware of that? KB: The old information with the permitted chars is still presented above the edit field. This should be updated. I also noticed one other very minor thing: two service type names are using small initial letters, they should be capitial. This is the case for the Registry and the Gateway as far as I remember. But this is an absolutely minor thing. DM: Ok. MB: Ok, I just keep it in the Actionpoint list, but this is practically solved. MB: Next actionpoint: The UNICORE probes integration ticket. Last update was that we missed the deadline for SAM Update-12 release missed. Emir, are we still in time now for SAM Update-13? EI: The integration of the UNICORE probes require a lot of changes from our side. The update... JG: You are a bit quiet, Emir! EI: Let me see, I'll try to increase the gain. MB: Now it is better. EI. Ok, I think I'll manage to integrate the probes by Update-13, but Update-13 has been pushed to August 12th. Actually it was supposed to be end of July, but due to the holiday season it had to be pushed back to August 12th. MB: So you are saying the deadline for the whole Update-13 release has been pushed back to August 12th? EI: Yes, and since until then it is almost a full month period, by that time it should be integrated. MB: Ok, thanks for the update. MB: The next actionpoint refers to accounting. I hope I wrote a correct update to it last time. That the important deadlines were end of June and end of July and that a wiki should have been created and and a discussion group. And John Gordon will make workshops at the Technical Forum if I understood, to discuss the details on how to best integrate the current UNICORE accounting solution. John, I'm sure you would like to add something. JG: The wiki exists, a mailinglist was created. Krzysztof is a UNICORE person on it. I don't remember, I would have to check if there are any other UNICORE representatives. MB: Can you post a link please to the wiki and the mailinglist? JG: Yes, sure. MB: And you already contacted Krzysztof then over this mailinglist on his input for the workshops then at the Technical Forum? JG: I am not going as far as the Technical Forum workshop, yet. We hope that people will have a look at it and give their feedback until then. Whether it is possible from what they have done already. Whatever. MB: What is then the expected outcome of the workshops at the Technical Forum? JG: There are two workshops. One is about updates to the basic APEL. If you have gotten a new client until then to tell sites what is needed to migrate. For the other systems we' ll give some feedback and report and progress of that. Because SGAS and DGAS for example publish their records directly to the database, we need to move them to messaging before we can switch to the new production system. So the new production system has all this implemented by messaging. We run the old and new together from the clients. When people migrate the client the move for example from the old method of messaging to the new method of messaging. To get an integrated system with the two databases linked together we need to move the direct database insert clients like SGAS, DGAS, Gracia. They are the ones that we need to move, because they are already doing it by a particular route. But anybody that isn't currently publishing like UNICORE, Globus,.. they can be part of this mailinglists, wiki and take part of the experience of others. MB: So UNICORE and Globus could already try against this new test system that you mentioned. JG: Yes, but the difference is: The others have done it already and know what data they need, how to do a database insert of what looks like a Usage Record. So they do essentially the same thing to the same database, they are taking the records and put it in a different messaging format using STOMP and send it to a different server instead of running the database insert. They all understand what they are doing. I think with Globus and UNICORE we need more of a one to one phoneconference with them, just to talk this through. I suspect we can do something on the wiki and the mailinglist, but a one to one meeting to kick things off and clear some of the details. So the question I have and probably asked it before. Is there actually a single UNICORE accounting service? Is it part of a release or something? Because the last time I heard about it, there seemed to be a Polish solution and a Belarusian solution RB: And there is a German solution as well. JG: Yes, right, we have somebody on the mailinglist from Hannover. MB: I think the Polish solution is currently the supported one. JG: The German solution seems to be a bigger solution, it was planning to take gLite, UNICORE and as I understood possibly Globus as well in a single NGI repository RB: Yes, that's correct. The D-Grid, the German NGI solution covers all middleware. KB: But, I think, the answer in the case of UNICORE is that in general there is no official UNICORE blessed accounting solution, which is officially supported by the core UNICORE or EMI, at least I don't know of such a thing. MB: I was just assuming that we go for the Polish solution, but if you say that the D-Grid solution is another option, we can discuss this here. This is what our task force is for. We can decide what we can go for in the future. JG: So the German solution is as I thought acting as a repository gathering information from different things. So they just need a UNICORE client for accounting. MR: That's right. They collect the data and it gets extracted tools from the local accounting. It is not a UNICORE solution as such. As Krzysztof said there is no real in fully integrated UNICORE accounting. The infrastructures, the national ones have their solutions for their purposes. Check which ones suites EGI purposes best and try to reuse things which are there. TF: Maybe this could be the topic for the next UNICORE meeting to have a functional comparison between the two solutions with each other and see how these complement each other, if they do and how flexible they are. JG: You could argue that if you have a system that is not UNICORE specific, if you've got a system that is just gathering accounting information you could presumably install more or less any accounting system. There a just a few things that tie it to UNICORE, that is my understanding of services. I'm sure that is the case. I was planning to have a meeting with the Hannover people, because they have a role in JRA1 as well. TF: When do you plan to have this meeting. JG: Next week or the week after. MB: Then you'll just be inviting the people from Hannover or also e.g. Krzysztof? JG: For the moment I just want figure what the people in Hannover are doing and what assumptions can be made. Once we have that and discussed the wider issues, we can invite all the people and use one of these UNICORE integration meetings to have a longer accounting agenda item. MB: The next actionpoint is also tied to accounting, the accounting solution of Belarus. Tiziana contacted the responsible director Andrew mentionened and the response was that it can't be legally made so easy open source, because it was bound to the funding by Belarus and Russia within the SKIF-GRID program and if we don't put in our own funding, we don't have chances to make it openSource. It is unlikely that any EGI-InSPIRE funding can be allocated to this project, as all JRA1 funding goes to the maintenance and development of central common tools, and the PMB has decided to allocate the small pocket of unspent effort accumulated during PY1 to sponsor travelling of smaller partners. So currently I don't see any money in the near future for this within EGI. Maybe if it would be good if they tried to get money somewhere else, e.g. within the UNICORE community. JG: If the Polish solution is good enough, why bother? MB: I still think we could profit. We are in an international project and should try to collaborate together to find an optimal solution. But you are right: the Polish solution improved a lot recently and we are quite satisfied with it. JG: I'm just saying, if we are going to start to actually buy a middleware that would be a dangerous street to go down. We have to be careful what the other applications are of all the other middleware providers. MB: That is in each case the situation at the moment. I don't know, Andrew, would you like to add something? AL: I have the same information as you cited above, I just received a copy of the communication between Uladzimir and Tiziana and don't have further information. MB: I think I would vote for closing this action point now, because I don't see that we can do anything at the moment. I'm sorry. MB: The next action point was about Best practices. I still haven't heard anything from the operational documentation group. I think I should contact them again. They had their meeting in Switzerland last month. I update the actionpoint that I should contact them again. MB: Then next actionpoint: List of servicetypes to be integrated. We had this discussion last time and we put togehter a list, according to that I updated the ticket. And I have seen David has already updated the description in the wiki: [10:24:33] Michaela Barth https://wiki.egi.eu/wiki/GOCDB/Input_System_User_Documentation#Service_types MB: any more questions, David or you also think we can close this actionpoint now? DM: That's fine from my perspective, if you need further unicore6 service types besides the one mentioned in the ticket just let me know [10:25:11] david meredith https://rt.egi.eu/rt/Ticket/Display.html?id=944 DM: We mentioned other service types like the unicore6.GridBeanService KB: We are okay for the moment with the current set of services. MB: David, you weren't here last meeting, when we had this discusion. We are fine with the current set of services for the moment. MB: Then I can close this actionpoint as well. KB: Yes, it is okay to close it. MB: The next actionpoint was the doodlepoll for this meeting. Then we had a maybe actionpoint that Krzysztof would create an RT ticket about the integration of NGI specific service types into the regional dashboard. Have you done something, Krzysztof, just out of interest. KB: This was just out of curiosity that I asked for this. But I think Marcin or someone else has created such a ticket and it will be handled in the next OTAG meeting. MB: The one now, next Wednesday? KB: Yes, but it is not really my business. MB: Yes, it is a little bit of our scope here, this is why it is in brackets. So but then I'll just close it. MB: 3) Discussion of Progress: a) Accounting MB: You haven't posted the link to the wiki and the mailinglist, yet. JG: Sorry about that. But I don't think I have anything else to add. I think we already covered it all in the actions. MB: So you will contact the people, you will organize the workshops. JG: Yes. MB: b) GOCDB MB: We have tried to enter some services, and we have now at least 2 sites within GOCDB. Did you see any more problems? MR: Not at the moment. MB: c) Monitoring MB: We already discussed that SAM Release 13 has been pushed back to 12th of August. Emir, you want to add something. EI: No, that's it from my side. MB: d) Staged roll-out MB: I'm not quite firm on that right now. The verification phase was passed and Rebecca, you wrote that all the packages in the staged-rollout have been tested. How was the procedure, was everything clear? RB: Yes, yes. During the verification we had to start from scratch for UNICORE, but next time will be easier, I hope. MB: So now it is basically through staged rollout, right? RB: Yes, everything is in the UMD release 1.0 from this week. MB: Yes, I have seen a long list of UNICORE services there. MB: Will you then take over the position of partial staged-rollout manager as we have it for ARC? Or was this already decided. Mario David has his submanagers for the different middlewares. He is doing the staged-rollout manager for gLite, for ARC there are two other persons doing it to keep things going and have a little bit better understanding of the middleware. Did he contact you and ask you to be that sub staged rollout manager? RB: No. TF: Michaela, I think there was a discussion in the OMB in April and the managers for ARC and UNICORE were already listed there. So we are done there, so I don't think we have any issue with that. I'll post it in the chat window. [10:34:00] Tiziana Ferrari UMD information is available at https://wiki.egi.eu/wiki/Middleware#Unified_Middleware_Distribution MB: Thanks. MB: e) Argus Krzysztof, do you have any news on Argus? KB: Not much, we are just about to perform a very small update of our Argus callout to the EMI common algorization profile Unfortunately this work is quite theoretical, because Argus will also include then this profile in late Autumn. I think we can close this point the next few months, because the message will be the same. Probebaly in one month time we'll be finished and will include everthing in testing, we wont to be able to make any further progress. MB: How will the status then be with UMD1.2? KB: I have no idea. MB: The first update I meant. KB: I can only say about updates in EMI and we have already had our update, and no feature changes in Argus won't be included before EMI2. we have this feature working area to perform integration test, but officially it will be out in EMI2. MB: So we don't expect any changes there, ok. KB: Can you say, or maybe Tiziana, what is the normal delay between an EMI update and its inclusion in an EGI UMD release. TZ: The delay depends on the criticality of the update. So if this is an intermediate release which is highly critical, the verification and staged-rollout phase can be very short. For instance one day. So in this case the gap , the lag between the EMI and the UMD can be very small. If the update has not such a high priority, you need to accomodate some time for verification and staged-rollout of the different components which can be different for some components. Typically the staged-rollout takes a couple of weeks. The timing really also depends on when the packages were ready. Typically there is no information in advance when an EMI update is due. For UMD we have a predifined schedule, so UMD1.1 will be out first week of August and depending on the relative timing an EMI release will be taken into account by an UMD update or not. So if an EMI update was released a week before the UMD update, the time won't be enough to have it for this update and you will have to wait for the next one. Following a request from the NGIs subsequent updates for a given component should not happen too often, so they should be consilidated UMD update. KB: Thanks for the explaination. MB: Yes, thanks Tiziana. MB: f) Best Practices MB: Ok, I don't think we have an update there. MB: 4) Changes needed to OLA and site certification procedure OLA: https://documents.egi.eu/secure/ShowDocument?docid=31 site certification: https://wiki.egi.eu/wiki/PROC09 MB: We had a discussion on the mailinglist, where Krzysztof brought up that we might have problems with 10.3 in the OLA and also with some points concerning the the BDII in the site certification procedure. What we should discuss is if we have a better suggestion so we then can ask the OMB to change these documents and make them more middleware independent and really suits our case as well. [10:37:36] John Gordon II http://dl.dropbox.com/u/9548018/APEL%20SSM%20background.txt http://dl.dropbox.com/u/9548018/APEL%20SSM%20testing.txt [10:37:56] John Gordon II These are mails explaining things. Links to wiki and mailing list in the mails. MB: Ah thanks, John! MB: Back to the OLA discusion. If you really see problems with the OLA we definetly have to propose changes to it that work with the current technical possibilities we have. It is hard for me to formulate something that really works, because my technical understand is maybe not big enough to formulate it in a way that still works for UNICORE. ..... [10:41:21] Michaela Barth you still hear me, yes? [10:41:43] John Gordon II no [10:41:44] Tiziana Ferrari no [10:42:03] Michaela Barth ok, strange, I'll log out and in... [some part of the discusion has been missed] KB: I'm not part of this development team, but I am pretty sure that this service is going to replace UNICORE registry, the UNICORE CIS and also BDII, etc. JG: The BDII in gLite is currently used for two different things. It is used for service discovering, so for pretty static things, like which endpoints are out there and that functinality can be presumably taken over by the EMI registry, but it is also used for dynamic things like publishing the load of the system and the service discovery is not going to do that. So that would still be needed. That depends if your information system does that already. --> AP on Tiziana to bring site-information (BDII) discussion up at the TCB --> AP on Michaela to formulate a more highlevel and middleware independent version of point 10.3 in the OLA and create a new ticket in the noc-managers queue for it and assign it to Tiziana, so it can be discussed at the next OMB 5) Input needed for MS414 Integrating Resources into the EGI Production Infrastructure https://documents.egi.eu/secure/ShowDocument?docid=650 6) Taskforce leadership handover Gert Svensson gert@pdc.kth.se will take over from 1st of August. 7) Next meeting a) fixed at the EGI TF, Lyon, week 38 8) AOB [10:44:19] Emir Imamagic sorry I gotta run [10:44:20] Emir Imamagic bye [10:44:23] Emir Imamagic left [11:08:06] John Gordon II Going now. Bye [11:08:16] John Gordon II left [11:10:42] david meredith goin too, bye [11:10:48] david meredith left [11:12:36] Mathilde Romberg left [11:12:36] Krzysztof Benedyczak left [11:12:38] Rebecca Breu left [11:12:41] Andrew Lukoshko left [11:12:49] peter solagna left