Speaker
Wider impact of this work
Data access on the grid has traditionally meant custom protocols and custom software blocks. The work presented here tries to break with this tradition, by enabling access using standard protocols and opening DPM for integration with standard building blocks both for frontends and storage backends.
The goal is to reduce the dependencies on custom software and reuse as much as possible the existing open source tools fulfilling the same requirements, such as Apache2, Memcache, Hadoop, S3 based cloud providers, among many others.
Description of the work
HTTP is the most widely used web access protocol. With DAV, it provides a standards based mean of accessing data on the grid, a valid alternative to traditional grid protocols like GridFTP.
NFS is a popular remote access protocol. With version 4.1 it introduces sessions, strong authentication, bulk operations, among many other features. With the addition of pNFS it also separates metadata from physical data access, while enabled parallel reading and writing of the data. This puts it as a very strong solution for POSIX access to data on the grid, offering additional benefits like free clients and kernel level caching.
DPM now provides frontends exposing both of these protocols, and we have been working on providing excellent performance and validating our software against popular scientific frameworks like ROOT or experiment specific software.
At the same time, and triggered by some performance limitations found while evaluating the analysis use cases against DPM, we have done a significant refactoring of the software. The result is a new plugin based library called 'dmlite', which exposes interfaces for namespace management, pool management, and access to data.
We present the global architecture of the new libraries, the plugins developed and the ones in development. These include memcache for scalability and excellent performance of the nameserver, hadoop as an alternative solution for pool backends, S3 for extending DPM storage to the cloud, among others.
We also present extensive performance results of each of these parts of DPM, including comparisons against previous setups where appropriate.
Printable Summary
The Disk Pool Manager (DPM) is a lightweight solution for grid enabled disk storage management. Operated at more than 240 sites it has the widest distribution of all grid storage solutions in the WLCG infrastructure. It provides an easy way to manage and configure disk pools, and exposes multiple interfaces for data access (rfio, xroot, nfs, gridftp and http/dav) and control (srm). During the last year we have been working on providing stable, high performant data access to our storage system using standard protocols, while extending the storage management functionality and adapting both configuration and deployment procedures to reuse commonly used building blocks. In this contribution we present extensive performance results of the work recently accomplished, we summarize bottlenecks discovered during this analysis and the solutions followed and we present new components opening DPM to integration with other popular storage systems.
