Powered by Indico
v3.2.7
Minutes initially generated by Zoom AI companion, then reviewed by Catalin Condurache and Gergely Sipos (EGI F.)
The team discussed updates to the accounting portal database, with a focus on incorporating environmental metrics and improving data granularity. They also explored the development of a metric system to measure the environmental impact of computing and the potential of an API for posting extra metrics to the portal layer. The conversation ended with plans to set up a test repository, further explore the use of an API, and aim for a solution within two weeks.
Catalin initiated a meeting to discuss updates to the accounting portal database, similar to the hep score. Adrian provided feedback, but Carlos was absent and needed to consult with the Quanta people. Ivan clarified the distribution of responsibilities between CESGA and Qantas concerning the EGI, with CESGA handling contracting and Qantas subcontracting for development activities. Cecilia emphasized the importance of Quanta's continued work with the accounting portal.
Gergely discussed the development of a metric system to measure the environmental impact of computing in EGI and other digital services. The system would gather data on energy consumption and carbon content to calculate the carbon footprint of computing. Gergely presented a high-level model for EGI, which calculates energy use from dynamic information in the accounting system and incorporates a static number specific to each hardware element. The model was designed to be implemented in the EGI system, with the aim of enriching the accounting data with power consumption and usage efficiency. The ultimate goal is to calculate the kilowatt hour consumption in a given virtual organization.
Gergely discussed the need for an estimation of the time required to introduce an additional benchmark for energy consumption accounting in the accounting repository database and the accounting portal. Ivan suggested that this could be achieved by building new systems around the existing accounting pipeline without requiring significant development effort. He also proposed the idea of incorporating new components and capabilities within the accounting portal, potentially using modern technologies. Adrian clarified that there is no API currently available for querying the repository or the portal, and that data is pushed to the portal. Ivan suggested two options for implementing an API: either pushing data into the APEL database and then querying it, or directly posting data into the API. He estimated that the development of this API could be completed within a few months, depending on the resources allocated.
Adrian suggested providing an API for posting extra metrics to the portal layer. Catalin raised concerns about the technical feasibility of this approach, particularly regarding the calculation of PowerHEPscore. Ivan proposed that they could query and calculate these metrics in their new API. Gergely expressed a desire for fine-grained data. Adrian proposed a model where submit hosts could be matched to the CEs in GOCDB, and Catalin agreed to explore this further.
Adrian and Alessandro discussed the complexities of benchmark normalization in different hardware environments. They noted that batch systems perform their own normalization locally, which can lead to complications. Alessandro explained that the monthly energy consumption metrics are calculated by associating jobs with specific hardware values and then normalizing them. Adrian added that the accuracy of these aggregates depends on the size of the aggregate. The team also discussed the challenges of tracking hardware diversity at the accounting repository level, with Catalin noting that some sites do not bother to create benchmarks for their clusters.
In the meeting, Catalin, Adrian, and Gergely discussed the limitations and potential of their current data collection system. They considered the possibility of obtaining more granular data, such as per-job energy footprints, but noted that this would depend on the sites' configurations. Adrian mentioned that some sites send individual job records, while others only provide aggregate data. The team also discussed the idea of extracting historical data for case studies, but acknowledged that they might not have information on the hardware used in the past. Gergely emphasized the need to understand the level of granularity they can achieve to define realistic views. The conversation ended with the understanding that the system's capabilities would depend on the sites' data submission practices.
Adrian proposes to investigate the split between jobs and summaries for the sites and provide sample data to give an idea of the structure. Gergely emphasizes the importance of obtaining job-level and site-level statistics as real-time and fine-grained as possible for brokering future jobs. They discussed the current limitations of the accounting workflow, which can take up to two days for job data to appear in the portal, and consider how this information could guide new providers in defining more real-time endpoints.
Gergely and Ivan discussed the development of a system for environmental metrics publication. They agreed on two main actions: Ivan to prepare a proposed architecture for the system and Adrian to subset data for further specification. They also discussed the need to minimize development on the existing EGI system side and the possibility of funding in a future project. The overall objective is to create a system that takes the books for the current project and serves as a starting point for something bigger.
In the meeting, Gergely discussed the expansion of the metric system to include various types of services, such as HTC, cloud computing, and specialized storage systems. He emphasized the need for these resources to report into the metric system for overall workflow optimization. Adrian proposed a separate test setup for cloud to see the workflow if the green metric was injected into the accounting record. However, he expressed concerns about affecting the benchmark in production. Catalin suggested that this could be a possible option, but it would require a separate test setup on Adrian's side. Ivan agreed to work on the slides and catch up with Cecilia and Carlos. The team also discussed the need for documentation on how to generate new accounting records for new types of resources.
Adrian, Gergely, and Catalin discussed the possibility of setting up a test repository to handle additional metrics. Adrian suggested that this could be done relatively easily, but the development of a system to display the data would require more effort. Gergely proposed using the test repository for all equipment, including WeNMR, Biomed, and future ones. The team agreed that the test repository could be used to exchange metrics directly, rather than using GOCDB. However, they also acknowledged that this would require more time and effort. The team decided to proceed with setting up the test repository and to explore the possibility of integrating it with the new service.
Catalin proposed to look at a test repo and consider potential future involvement of EGI Federation members in calls. Adrian agreed to think about providing access to the test repo and mentioned that the test repo would only use existing schemas and workflows. Gergely suggested that they could build everything around the data they receive from Adrian's system, as long as they can get the job level metrics and correlate them with the green energy efficiency of the site. Adrian clarified that the test repo would still receive a single benchmark, whether it's cloud or grid, and that it wouldn't be a way to solve the overall problem of the project. They agreed to continue with the test repo and explore its potential for accommodating other types of requests from non-HTC.
Gergely, Adrian, Catalin, Cecilia, and Alessandro discussed the implementation of a test bed for their project. They agreed to aim for a solution in two weeks, with a meeting scheduled for the 16th of April at 11:30 Central European Summer Time. Cecilia offered to assist with a document related to the project's architecture. They also discussed the possibility of using an API to query data directly from the portal, with Gergely expressing interest in this approach. The team agreed to further explore this option.
A follow-up meeting was scheduled for two weeks time - Wed 16 April 11:30 CEST