EGI Community Forum 2014 - Helsinki

Application management framework to dynamically exploit grid and cloud computational infrastructure: JST and DIRAC.

22 May 2014, 14:50

25m

Room 6 (Helsinki University, Main Building)

Sessions contributions Virtual Research Environments, gateways and workflow engines (Track Leaders: J. Montagnat, G. Sipos) DIRAC Virtual Research Environment pilot for EGI

Speaker

Dr Giacinto Donvito (INFN)

Description

Nowadays, the execution of a huge number of concurrent applications is a common problem for several small and medium users communities (Bioinformatics, Biomedicine, astrophysics, etc). Moreover, the available computational technologies able to exploit both grid and cloud infrastructures, require a step forward in terms of abstraction and flexibility of the tools used to manage application submission. This work presents a detailed comparison in terms of functionalities and performance between JST and DIRAC, which provide standard web services interfaces (REST or SOAP). In particular, capabilities of such tools are described, in order to outline the possibility to submit job from standard high level application, such as workflow management systems (Taverna, LONI pipeline, Galaxy), exploiting different kind of computational resources in a seamless way. Moreover, the work conducted in order to exploit both JST and DIRAC within standard web portals technologies like Liferay is discussed.

Description of work

The Job Submission Tool (JST) is a homemade solution developed since 2005 by our research group in order to deal with job submission, monitoring and resubmission based on the concept of pilot jobs and task queue.
 Nowadays, this tool is able to exploit different kinds of computing technologies: grid resources, cloud IaaS resources, local batch cluster and standard dedicated servers. JST also offers two web services interfaces (based on REST and SOAP technologies) to submit job from standard high level applications as workflow managers and web portals (Taverna, LONI pipeline, Galaxy, Liferay, etc). It is already used in different projects to support grid activities of many communities (Bioinformatics, Biomedicine, Astrophysics, etc).
DIRAC is a framework widely used to exploit different computational resources in terms of both computing and storage. It is already used from several scientific communities such as LHCB, biomed, Belle II, etc., and includes a high number of functionalities such as pilot job submission, file and metadata catalogue, automatic software installation, etc.
 The comprehensive tests conducted to evaluate the performance of those two tools are described and, in particular, the experimental evaluation is focused on the detailed analysis of the provided functionalities, in terms of both workload and data management solutions.
The last part of this work contains a detailed description of the effort needed to use JST and DIRAC tools as back-end for other workflow managers (Taverna, LONI pipeline, Galaxy,), underlining the possibility to use such two tools to simplify the development of high level Web user interfaces. For both these last use cases, the way to successfully exploit web services interfaces of JST and DIRAC is shown.

Wider impact and conclusions

Both JST and DIRAC provide high level and user-friendly interfaces able to hide the complexity of the underlying computing infrastructures so that end-users, workflow managers and web portals, can easily access recources provided by different platforms: grid computing, IaaS cloud infrastructure, local batch farm or dedicated servers. Within this work, the differences among JST and DIRAC are highlighted, in order to simplify the choice of the tools that better fit the scientific communities and end-users requirements.
The possibility of exploiting heterogeneous computing infrastructures in a seamless way through a workflow manager improves the user experience. In fact, those tools are able to manage the execution of applications on a distributed computing infrastructure by reducing the overhead for the end-user.

Presentation materials

Slides

JST-DIRAC_Donvito.pdf