Speaker
Impact
Service grids like the EGEE grid can not always provide the required number of resources for many VOs. Therefore extending the capacity of these VOs with volunteer or institutional desktop grids would significantly increase the number of accessible computing resources that can particularly advantageously be exploited in case of parameter sweep applications.
EDGI is currently focusing on three different areas: one is to create technical solutions and develop the necessary software components gLite, ARC, UNICORE, Attic, 3GBridge, Application Repository, Monitoring, Boinc, XtremWeb, Eucalyptus, Opennebula to enable the creation of a Desktop Grid to support Service Grids. The second main area is to continuously operate and maintain the EDGI infrastructure built by these software components and provided as a production service for any EGI User communities. The third main area is to extend the capacity of the infrastructure by institutional and volunteer desktop and cloud resources to continuously keep up with the requirements of computational capacity.
In order to provide sustainability for the project, EDGI together with its neighbour project called DEGISCO have recently established an organisation called International Desktop Grid Federation (IDGF) which aims to bring together people (technical experts, developers, operators, users or any interested people) from all areas related to desktop Grid computing and aims to share the common knowledge among its members.
URL
http://edgi-project.eu
http://desktopgridfederation.org
Conclusions
The EDGI Desktop Grid and Cloud infrastructure contains a large number (>100.000) of volunteer and institutional desktop resources. The EDGI infrastructure contains bridging services which are able to automatically forward jobs from Service Grids (currently gLite and ARC) to Desktop Grids (currently BOINC and XtremWeb). Moreover, Cloud resources (currently Eucalyptus/Amazon) can also be integrated to expand the currently available reliable and non-reliable resources based on the latest developments.
EGI User communities can gain significant advantages of this huge pool of resources or if there is a need they can setup their own pool based on the EDGI software components. The solution brought by EDGI is also ensuring that the EGI users do not need to change their well-known environment, since the infrastructure developers provided the access of the desktop grid and cloud resources through gLite and ARC user interfaces. Within a year the Unicore interface will also be supported.
Description of the work
The SG to DG bridging service which is being developed by the EDGI project lean on two key components: modified computing element on the service grid (gLite, ARC, UNICORE) to forward the job to a desktop grid site and the 3GBridge service on the desktop grid site which is able to receive and transform the job as a workunit for execution for the desktop grid (BOINC, XtremWeb) site.
The objective of automatically forwarding gLite jobs to desktop grid resources has been already achieved by the former EDGeS project that built a production infrastructure enabling the extension of gLite VOs with several volunteer and institutional desktop grids. However, this version had some scalability issues related to the gLite modified CE and to the fact that all data have been forwarded through the bridge components which became a bottleneck.
In EDGI one of the focuses is on eliminating scalability issues, thus first we changed gLite CE to CREAM CE, secondly we added URL pass through mechanism to the 3GBridge software in order to be able to forward the reference to files instead of the file itself. Moreover, attic (a P2P file system) has also adapted to the EDGI bridging components to increase the efficiency of data transfer among the server and clients.
In order to seamlessly integrate Cloud resources into the EDGI infrastructure, we have extended the bridge with a special, automatic Eucalyptus Cloud handling mechanism, including instantiation and job execution. Due to the plugin framework of the 3GBridge this could easily be implemented by adding a new plugin without modifying the core software. With this extension jobs arriving to the bridge can be executed on Cloud resources.
Based on the solution above, the first software release of the EDGI project has been finished. Currently, we are working on integrating our desktop grid infrastructure — containing tens/hundreds of thousands of computers — together with our bridging services into the EGI grid infrastructure.
Overview
The EDGI (European Desktop Grid Initiative) FP7 project is aimed at deploying Desktop Grid and Cloud Computing services for the EGI research user communities that require large-scale distributed computing resources for multi-national projects. In order to achieve this goal EDGI develops middleware for extending Service Grids (ARC, gLite, Unicore) with Desktop Grids (BOINC, XtremWeb) enhanced by Academic Clouds (Eucalyptus and OpenNebula). Software components of ARC, gLite, Unicore, BOINC, XWHEP, Attic, 3GBridge, OpenNebula and Eucalyptus will be integrated into a SG to DG to Cloud platform for service provision and as a result EDGI will extend ARC, gLite and Unicore grids with volunteer and institutional DG systems. In this way, the whole European e-science ecosystem will benefit from Desktop Grid extensions, since parameter sweep applications that run millions of sequential jobs can be directed from the expensive cluster and supercomputer resources to cheap desktop resources.
