9-12 May 2017
Europe/Rome timezone

Dynamic On Demand Analysis Service : DODAS

Not scheduled
15m

Speaker

Daniele Spiga (INFN)

Description

We describe the implementation of an automated system that simplifies the process of provisioning, creating, managing and accessing a pool of heterogeneous computing and storage resources, by generating clusters on demand to run batch systems, thereby implementing the “Batch System as a Service” paradigm. We also detail how such service has been seamlessly integrated with existing computing infrastructure of the Compact Muon Solenoid (CMS) experiment as a solution to provide additional (e.g donated, hired, opportunistic, etc...) computing capacity to the collaboration. The architecture of the described service is built using several technological solutions developed in the context of the INDIGO-DataCloud project: the PaaS Orchestrator and the Infrastructure Manager together provide the abstraction layer for the definition and the provisioning of virtual platforms to guarantee a transparent exploitation of multi-backend (private and public IaaS clouds); the Identity and Access Manager, supporting federation through multiple methods (SAML, OpenID Connect and X.509), provides a flexible platform to securely integrate community specific credential services. The automation strategy is also implemented adopting INDIGO-DataCloud solutions and thus the cluster topology, as well as the orchestration of all services such as HTCondor, Mesos, Marathon, Squid proxy, including any possible software dependency, are orchestrated using TOSCA templates and Ansible roles. A key outcome is that a single YAML file describes any complex setup leaving the end-user just with the management of a trivial configuration file. Apache Mesos is used to manage CPU, RAM and storage offered by cloud providers, while Marathon is the adopted application framework. The Batch System is based on HTCondor technology.

Primary author

Daniele Spiga (INFN)

Co-authors

Andrea Ceccanti (INFN) Dr Cristina Duma (INFN-CNAF) Davide Salomoni (INFN) Dr Giacinto Donvito (INFN) Marica Antonacci (INFN) Tommaso Boccali (INFN)

Presentation Materials