Powered by Indico
v3.2.7
Minutes initially generated by Zoom AI companion, then reviewed by Catalin Condurache and Gergely Sipos (EGI F.)
Quick recap
In the introduction Catalin presented the objectives and challenges of the GreenDIGIT consortium, focusing on lowering energy consumption and environmental impact, and proposed a new benchmarking system for grid and cloud accounting records. They also explored the possibility of modifying an existing accounting client to meet the needs of GreenDIGIT. The participants also agreed the benefit of a workshop for aligning priorities with WLCG and EGI for the future of accounting in general, and scheduled a follow-up meeting to discuss outstanding work items.
Actions
1. Catalin to check with the cASO team if additional benchmarks can be added to the cloud jobs records and if sending them to the accounting repository would break it or not, and what updates are needed (if any).
2. Catalin to check which accounting clients are used by HTC sites that support WeNMR and biomed VOs
3. EGI to get in touch with developers of above clients and discuss how much effort is needed to add capability to support additional benchmarks in records
4. EGI to investigate the suitability of AUDITOR tool (https://alu-schumacher.github.io/AUDITOR/latest/) to be used for the GreenDIGIT purposes primarily on sites from other RIs (non HTC and non cloud resources)
5. Adrian/STFC to provide an estimation within a week of the work required to extend the Accounting Repository database to be able to accept one (or more) dditional benchmarks for energy accounting.
6. Carlos/CESGA to evaluate how much effort would be needed to extend the Accounting Portal with new views based on new ‘green’-related benchmarks (i.e. energy and CO2 based)
7. Catalin/EGI to check which VOs are of interest for Cloud (WeNMR and biomed already candidates for HTC)
8. Catalin/Gergely to schedule a follow-up meeting in two weeks to discuss priorities and roadmaps for the EGI Core Services. (2/Mar, 11:30 CEST)
Summary
GreenDIGIT Consortium Energy Consumption
In the meeting, Catalin presented the objectives and challenges of the GreenDIGIT consortium, focusing on lowering energy consumption and environmental impact. The consortium is particularly interested in measuring energy consumption and carbon footprint during the operation phase of the research infrastructure. Catalin presented a formula for calculating carbon footprint, which involves energy consumption, utilization, and carbon content. He also discussed the proposal from CESNET to implement green resource ranks and carbon intensity as dynamic site attributes. The conversation ended with a discussion on the proposed implementation, including the use of GOCDB, ongoing development from CSIC, and the potential for green-specific algorithms in job brokering.
Handling New Benchmark in Accounting Record Format
Adrian discussed the accounting record format (sent through the Messaging service), which can accommodate arbitrary benchmarks. However, the system currently only prioritizes the primary benchmark, such as HEPscore23. If a new benchmark is introduced, it would be ignored. The development team needs to decide how to handle and store this new benchmark. Alessandro suggested that sites providing ARC CE should set this new benchmark in their configuration files (this applies in fact to any CE type). Catalin proposed replacing the SPEC CPU2017 with the existing HEPscore23, divided by TDP for the processor, to simplify the task for sites to compute the GreenResourceRank benchmark (as CPU HEPscore23 / CPU TDP)
New Benchmark Implementation in Accounting Client
The team discussed the implementation of a new benchmark in the accounting client. Adrian clarified that the accounting client wouldn't know what to do with the new benchmark, and it would need to be generated and consumed. Gergely asked about the client used by CSIC (ARC - Jura?), CNRS (ARC - Jura?), and CESNET (HTCondor - APEL?). The team discussed the possibility of different accounting clients handling multiple benchmarks and the need to check case by case (for ex. INFN use their own system to send records to accounting repository). They also discussed the need to identify the contacts for the different accounting clients and the potential for the new benchmark to be ignored at the repository level. Gergely proposed two steps: checking which accounting clients the sites are using and getting in touch with the technology providers to understand how the new benchmark would be handled.
Implementing New Benchmark and Client Modifications
In the meeting, the team discussed the implementation of a new benchmark, with Alessandro suggesting that the metrics should be agreed upon before deciding how to handle the additional benchmarks. Adrian and Catalin discussed the use of HEPscore23 as a benchmark, with Adrian suggesting that the green resource rank should be versioned (i.e. GreenResourceRank23, GreenResourceRank25 etc). The team also discussed the need to modify the accounting clients to handle the new benchmark and agreed to check which clients are used at various sites. Gergely raised a question about how to get the green resource rank values into the logs generated by the batch systems which the accounting clients parse to extract the data. The team also discussed the need to modify the existing software to handle parallel benchmarks.
The team explored the possibility of using existing tools like BDII for publishing 'green' benchmarks information that can be queried by tools like DIRAC to produce further resource ranking.
Implementing Benchmarking System for Cloud Accounting
The team discussed the implementation of a new benchmarking system for cloud accounting records. At the moment there is no benchmark used by the Fedcloud sites.
Calculating CO2 from Energy Consumption
In the meeting, Gergely and Catalin discussed the need for an accounting client to read and process data. They considered how to calculate the amount of CO2 generated by a VO, which is calculated by multiplying the CPU hour consumed with a static metric. They also discussed the need for the accounting portal to display this information once these specific 'green' metrics are defined.
Adrian asked whether the accounting portal would need to pull in some of that information to generate totals of CO2. This is expected to be provided by tool developed by CSIC in a separate WP of GreenDIGIT (as alternative to querying the electricitymaps.com portal). Alessandro added that indeed the accounting portal combines the accounting and the CO2 data to display consumption results associated to related 'green' metrics.
The team agreed that the accounting portal should be able to calculate the energy consumption of a job based on its duration and the green resource rank, and then an external system could multiply that with the CO2 content of energy.
Second Benchmark for Energy Accounting
Adrian and Gergely discussed the possibility of adding a second benchmark for energy accounting in the accounting system. Adrian suggested that it might be simpler now that some changes have been made, but he would need to consult with his team to determine the scope of work involved. Gergely asked for an answer within a week, while also investigating the accounting repository situation. Tom suggested meeting with the team to estimate the resources needed. Gergely emphasized the critical nature of this element in the project and the need to consider it for all providers, not just EGI resources.
Modifying Auditor for GreenDIGIT Needs
Gergely proposed the idea of modifying an existing client to meet the needs of other RIs in GreenDIGIT, allowing for the generation of different types of records from antennas, IoT resources, etc. Alessandro and Adrian suggested that an existing client called Auditor (https://alu-schumacher.github.io/AUDITOR/latest/), developed by the German sites, could be a suitable candidate for testing in case existing EGI middleware is not used..
Exploring Baseline Data for Environmental Impact
Gergely discussed the possibility of using existing accounting data as a baseline to determine job durations and flop usage, and then transforming that into environmental impact through power consumption and CO2 emission. He suggested building everything into the brokering system, but acknowledged that this might not be realistic. Catalin mentioned that some sites measure total consumption and can estimate energy consumed by an experiment during a certain time. Greg confirmed that the GOCDB API can be accessed and updated programmatically. Adrian clarified that the summarized data in the portal is month-based, but some sites send individual job records which could be a period of up to 2 days. The team agreed to go back and think about the implementation, and to talk to accounting clients and colleagues about generating another view once the data is available in the repository.
STFC Project Budget and Development
The team discussed the ongoing project with STFC or CESGA, which is currently without a budget. They acknowledged the project's complexity and the need for further development work. The team also discussed the potential for alternative financing methods or integrating the project into the EGI Core Service activities. They touched on the need for storage and GPU accounting. The team also mentioned the WLCG workshop in May and the need to understand the scope of the project and budgeting situation.
Aligning Priorities with WLCG and EGI
There was no prior discussion between STFC/CESGA and WLCG/CERN about similar topics. Aligning the EGI and WLCG roadmaps for accounting (in the broader sense) would be very useful. EGI management is in contact with WLCG management about such a meeting.
Next meeting
A follow-up meeting was scheduled for two weeks time.