9–11 Oct 2018
Lisbon
Europe/Lisbon timezone

dCache: storage for XFEL scientific use-cases and beyond

10 Oct 2018, 14:30
15m
Auditorium JJLaginha (Lisbon)

Auditorium JJLaginha

Lisbon

ISCTE, University of Lisbon
Presentation Area 5. Digital Infrastructures for EOSC and/or EDI Data Management Services

Speakers

Mr Michael Schuh (DESY) Patrick Fuhrmann (DESY)

Description

The dCache project provides open-source storage software deployed internationally to satisfy ever more demanding scientific storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems. In supporting new communities, such as medical research, photon science/XFEL and microbiology, dCache is evolving to provide new features and access to new technologies. Whatever the use case, for federated storage to work well some knowledge from each storage system must exist outside that system. This is needed to allow coordinated activity. To support such scenarios dCache provides a stream of internally generated events. In this approach the storage systems (rather than the clients) become the coordinating service, notifying interested parties of key events. Storage events are also useful in other contexts: catalogues are notified whenever data is uploaded or delete, tape becomes more efficient because analysis can start immediately after the data is on disk, caches can be "smart" fetching new datasets pre-emptively and removing cached content when the source is deleted. In this paper we will present work done at DESY in building a low-latency, compute cloud facility for various XFEL workflows. This was achieved by combining dCache storage events with various Open Source projects, such as Apache Kafka, Apache OpenWhisk and Kubernetes. The resulting "serverless" cloud service is similar to AWS Lambda or Google Cloud Functions. It allows the infrastructure to deploy additional resources automatically, seamlessly scaling to match the demand.

Summary

At DESY we are building building a
low-latency, compute cloud facility for various XFEL workflows. This
was achieved by combining dCache storage events with various Open Source
projects, such as Apache Kafka, Apache OpenWhisk and Kubernetes. The
resulting "serverless" cloud service is similar to AWS Lambda or Google
Cloud Functions. It allows the infrastructure to deploy additional
resources automatically, seamlessly scaling to match the demand.

Type of abstract Presentation

Primary authors

Frank Schluenzen (DESY) Juergen Starek (DESY) Dr Marina Sahakyan (DESY) Mr Michael Schuh (DESY) Dr Olufemi Adeyemi (DESY) Patrick Fuhrmann (DESY) Paul Millar (DESY) Sibel Yasar (DESY) Mr Tigran Mkrtchyan (DESY)

Presentation materials

There are no materials yet.