Speakers
Mr
Michael Schuh
(DESY)
Patrick Fuhrmann
(DESY)
Description
The dCache project provides open-source storage software deployed
internationally to satisfy ever more demanding scientific storage
requirements. Its multifaceted approach provides an integrated way of
supporting different use-cases with the same storage, from high
throughput data ingest, through wide access and easy integration with
existing systems. In supporting new communities, such as medical
research, photon science/XFEL and microbiology, dCache is evolving to
provide new features and access to new technologies.
Whatever the use case, for federated storage to work well some knowledge
from each storage system must exist outside that system. This is needed
to allow coordinated activity. To support such scenarios dCache provides
a stream of internally generated events. In this approach the storage
systems (rather than the clients) become the coordinating service,
notifying interested parties of key events.
Storage events are also useful in other contexts: catalogues are
notified whenever data is uploaded or delete, tape becomes more
efficient because analysis can start immediately after the data is on
disk, caches can be "smart" fetching new datasets pre-emptively and
removing cached content when the source is deleted.
In this paper we will present work done at DESY in building a
low-latency, compute cloud facility for various XFEL workflows. This
was achieved by combining dCache storage events with various Open Source
projects, such as Apache Kafka, Apache OpenWhisk and Kubernetes. The
resulting "serverless" cloud service is similar to AWS Lambda or Google
Cloud Functions. It allows the infrastructure to deploy additional
resources automatically, seamlessly scaling to match the demand.
Summary
At DESY we are building building a
low-latency, compute cloud facility for various XFEL workflows. This
was achieved by combining dCache storage events with various Open Source
projects, such as Apache Kafka, Apache OpenWhisk and Kubernetes. The
resulting "serverless" cloud service is similar to AWS Lambda or Google
Cloud Functions. It allows the infrastructure to deploy additional
resources automatically, seamlessly scaling to match the demand.
Type of abstract | Presentation |
---|
Primary authors
Frank Schluenzen
(DESY)
Juergen Starek
(DESY)
Dr
Marina Sahakyan
(DESY)
Mr
Michael Schuh
(DESY)
Dr
Olufemi Adeyemi
(DESY)
Patrick Fuhrmann
(DESY)
Paul Millar
(DESY)
Sibel Yasar
(DESY)
Mr
Tigran Mkrtchyan
(DESY)