ASTRON/LOFRA<br>
M. Wise<br>
Micahel...<br>
Luciano<br>
<br>
SURFASara<br>
Coen, Anatoli<br>
<br>
EGI.eu<br>
Diego, Enol, Tiziana<br>
Susana<br>
<br>
INTRODUCTION<br>
presentation on the LOFAR Calibration Pipeline (Susana), the problem to address: 1. the pipeline to be executed in a distributed environment, the data cube can be processed in different threads with parallelism, 2. WM image allows complete control on what needs to be installed, the management of libraries and sw installation is difficult (LOFAR software needs many libraries, the use of prepackaged tools and libraries simplified this).<br>
<br>
Several tests where conducted. Parallelization was ok, sw distribution was also well managed, the problem faced is the storage and better solution for the many datasets is needed. Input data needs to be mounted in every VM processing the data. Currently NFS is used and it is not the best approach.<br>
Q. Is a cluster file system a solution? S. the volume needs to be shared across different cloud sites, the volume should be accessible through different sites. Problem: accessing data volume from remote sites<br>
<br>
OBJECTIVES<br>
<br>
MW 1.PROBLEM 1. Join up computing infrastructure and the LTA data ==&gt; open up the archive (active archive), to lower barriers in accessing the LTA data. Make it easier for the average user.<br>
Imaging pipelines, calibration pipelines<br>
<br>
2. PROBLEM. Use the distributed computing facilities to support external usage (Leiden, more ASTRON facilities to be procured)<br>
<br>
3. PROBLEM. Support easier software updates<br>
<br>
<br>
CS: LTA is based on dCache. Downloading from web browser not possible but rules could be changed<br>
Some data is public and some data is not public<br>
Data becomes public after 12 months ==&gt; Coen this is not implemented in LTA<br>
<br>
For public data users do not need to be linked to any project and no right to browse the directories, so data is not browsable<br>
<br>
(A) review of AAI infrstructure needed for the AAI data (LTA/SURFSara)<br>
(A) help in making data recalling from tape easier and more scalable<br>
<br>
A download server based on http is possible. Users can download from that machine, with username/passwd provided by LOFAR. The limitation is that it is only 1 Gbps pipeline and the bandwidth is too limited<br>
Data downloaded to UK and DE, the bottleneck is the local network, eg. the firewall (downloading multiple TBs is not feasible)<br>
<br>
(A) evaluate a organized T2 infrastructure for LOFAR<br>
<br>
CS: 11 PB of tape and disk, next to it 5,000-6,000 cores<br>
LOFAR sw is installed in the cluster<br>
10 TB of local scratch space<br>
Lack of documentation =&gt; organization problem<br>
<br>
MW: the scripts are not generic, they are specific to some use cases, phyton scripts<br>
<br>
(A) ASTRON. Provide a generic pipeline/capability for reprocessing of LTA data for average user<br>
<br>
Need to use the distributed computing facility and the software updates<br>
<br>
Is Docker supported? LOFAR sw packaged in containers, possible in the SURFSara cloud which provides an isolated system. Security problems in the grid.<br>
<br>
(A) Help from SURFSara/EGI in studying how to port software to Docker containers<br>
<br>
S: how are the local clusters connected to the &quot;t1 centres&quot;? How is the LTA data moved from LTA to the local clusters? data has to be locally copied through scp or other mechanisms, it is difficult<br>
<br>
Use Case 1 =&gt; we bring users to the archives (T1 sites)<br>
Use Case 2 =&gt; use of other computing facilities (T2 sites, currently totally not organized)<br>
<br>
CONCLUSION<br>
LOFAR to define priorities of activities