Speaker
Impact
The new developments in cloud, campus grid, and networking accounting are presented to a wide community. This will stimulate discussion and an exchange of ideas about the standard format for these types of data and further facilitate the sharing of collected information among Grid sites. Success with transitioning to a new Gratia-APEL interface is crucial to provide accurate accounting for the LHC VOs and the OSG. We still have concerns about some missing APEL features that are necessary for the completion of the project.
Description of the Work
After being used in a production environment for several years, Gratia has shown to be a robust, scalable, and extendable service. The system is capable of accepting new categories of data without requiring significant changes to the underlying software. Many Gratia probes, developed by various teams, collect data for batch systems, linux processes accounting, storage element capacity, and data transfers. Probes are uploading data from remote locations to a network of Gratia services.
This talk will describe the work that has been done to incorporate FermiCloud accounting into the generic Gratia framework. FermiCloud is an open-source private cloud that provides IaaS on demand to the Fermilab community. It is currently using Open Nebula for cloud management. Open Nebula stores in its database the current status of active Virtual Machines (VMs) as well as all historical information related to the state and state transitions of the VMs. This data can be extracted by using an API provided by Open Nebula. We have developed a new probe that extracts this information and sends it to the Gratia Collector using the Gratia Probe API. No changes were required in the Gratia core software and database schema to accept this data. The Job Usage Record structure, based on OGF standards and implemented in Gratia, has been populated by the cloud accounting probe. We were able to generate graphical reports without any changes in the report web-portal.
We will present the changes done in the condor probe to accurately record Campus Grids usage of the OSG resources and detect the workflow of campus jobs. We will also outline the modification required for the condor probe to collect network usage of jobs and the necessary changes in Gratia core software to accommodate the collection and summarization of the new data. In this talk, we will also cover the progress with the modification of the Gratia-APEL interface to use ActiveMQ instead of direct access to a database for data upload.
Conclusions
The Gratia framework can be extended to incorporate new categories of resources, such as network and cloud. The accurate reporting and validation of the accounting information between the OSG and WLCG are important aspects of the collaboration. We foster a dialogue on the means of sharing accounting metrics with other Cyberinfrastructure providers and are in discussion with XSEDE on providing Gratia information to a common XD Technology Auditing Service.
Overview (For the conference guide)
The Grid Accounting Service (Gratia) was developed as a collaboration between Fermilab, US CMS, US ATLAS, and the Open Science Grid and is in use across OSG, US LHC, Fermilab and University of Nebraska. Repositories are operated by several sites, including OSG Operations at IU and Fermigrid. The Gratia development group is in close collaboration with the OSG software, campus, operations and technology teams who work together on requirements, distribution, and support.
This talk describes several aspects of our most recent work. Among other things, we have been working on a prototype that uses the Open Nebula API to extract information about Virtual Machine states and resource usage in a cloud environment.
Another project that is currently in development strives to accurately record Campus Grids usage of the OSG resources.
In addition, we are working on modification of Gratia-APEL interface as well as researching a means of sharing some of the metrics with other grid infrastructures.
