Bulk data management system

The bulk data management system (BDMS) is a sub-system of CTAO computing providing the distributed archive for the (Petabyte) bulk data, stored and processed by four data centres. The BDMS is responsible for the archival storage of large-scale data products following the Open Archival Information System (OAIS), an ISO standard. This standard was first developed for space scientific missions and the INTEGRAL archive (developed at UniGe in 1995) was the first designed according to this model. The BDMS ingests the raw data produced by the ADH at the telescope sites, preserves the data in the CTAO data centres and makes the data available to the workload management system, processing the data to higher levels. The main functional units of the BDMS are ingest; distributed data management; archival storage; query and le access; and administration. The ingest component includes validation checks that verify the conformance of every le before ingesting them for preservation. It also extracts scientific metadata. The data management uses the RUCIO framework developed at CERN for LHC (ATLAS and CMS) data management and guarantees that data products are preserved at least in two distinct CTAO data centres.