Speaker
Overview
This demonstration is about to introduce the latest achievements the EDGI (European Desktop Grid Initiative) FP7 project provided recently. EDGI is aimed at deploying Desktop Grid and Cloud Computing services for the EGI research user communities that require large-scale distributed computing resources for multi-national projects. EDGI is developing middleware for extending Service Grids (ARC, gLite, Unicore) with Desktop Grids (BOINC, XtremWeb) enhanced by Academic Clouds (Eucalyptus and OpenNebula). Software components of ARC, gLite, Unicore, BOINC, XWHEP, Attic, 3GBridge, OpenNebula and Eucalyptus will be integrated into a SG to DG to Cloud platform for service provision and as a result EDGI will extend ARC, gLite and Unicore grids with volunteer and institutional DG systems. In this way, parameter sweep applications that run millions of sequential jobs can be redirected from the expensive cluster and supercomputer resources to cheap desktop resources.
Description of the work
Key components of the SG to DG bridging service developed by the EDGI project are as follows: modified computing element on the service grid (gLite, ARC, UNICORE) to send jobs to a desktop grid site and the 3GBridge service on the desktop grid site to receive jobs from any service grid site. The 3GBridge component is also able to transform jobs to workunits to be inserted into the desktop grid (BOINC, XtremWeb) server.
In this demonstration we are focusing on two important fields like solving scalability issues and to enable the cloud resources to be utilised by the desktop grid server. To cover the first, we changed gLite CE to CREAM CE and added URL pass through mechanism to the 3GBridge software. URL pass through means to forward file references (URLs) instead of the file itself. Files behind a web server, behind an attic (a P2P file system) server or behind a gridftp server can be defined as remote inputs associated to the jobs. In case more reliable resources is needed, EDGI provides an enhancement of the bridge by which cloud resources can be integrated on the fly during operation dynamically. This enhancement includes instantiation of a prepared VM containing a Desktop Grid client and job execution on this VM.
The aim of the demonstration is to show how potential EGI user communities can execute their gLite jobs on Desktop Grid and Cloud resources by using the normal gLite command line interface. The main phases of the demo are as follow:
1. short overview of the EDGI infrastructure
2. detailing the way how the JDL for gLite should be prepared and how necessary information can be collected
3. various job submissions to gLite (BOINC or Cloud as target, input files as attic/http/gsiftp reference)
4. monitoring of job state and retrieving the result
URL
http://edgi-project.eu
http://desktopgridfederation.org
Impact
There are three different areas EDGI is currently working on: one is to create technical solutions and develop the necessary software components like gLite, ARC, UNICORE, Attic, 3GBridge, Application Repository, Monitoring, Boinc, XtremWeb, Eucalyptus, Opennebula to enable the creation of a Desktop Grid to support Service Grids. The second main area is to continuously operate and maintain the EDGI infrastructure built by these software components and provided as a production service for any EGI User communities. The third main area is to extend the capacity of the infrastructure by institutional and volunteer desktop and cloud resources to continuously keep up with the requirements of computational capacity.
In order to provide sustainability for the project, EDGI together with its neighbour project called DEGISCO have recently established an organisation called International Desktop Grid Federation (IDGF) which aims to bring together people (technical experts, developers, operators, users or any interested people) from all areas related to desktop Grid computing and aims to share the common knowledge among its members.
Conclusions
With a large number (>100.000) of volunteer and institutional desktop resources provided by the EDGI Desktop Grid and Cloud infrastructure EGI users are able to execute parametric jobs significantly more efficiently. The EDGI infrastructure contains bridging services which are able to automatically forward jobs from Service Grids (currently gLite and ARC) to Desktop Grids (currently BOINC and XtremWeb) or to Clouds (currently Eucalyptus/Amazon).
It is important to note that services and software components demonstrated together with the large number of attached desktop grid resources are available for any EGI user communities from now on. There are different ways of utilising the EDGI results: EGI user communities can setup their own desktop grid server, collect their own machines as resources in their institution and deploy the bridging services. Others may like to simply get access to the already established and operated EDGI production infrastructure.
