Array calibration and data quality pipelines

CTAO aims to provide scientists with the highest quality data to meet its ambitious science program. High accuracy in measuring the energies and directions of  primary cosmic particles requires an excellent understanding of the detector and the environment. The CTAO data processing system foresees two key subsystems to achieve the desired performance: a calibration subsystem of the DPPS called CalibPipe and a Data Quality pipeline for the assessment and monitoring of the data flow. As the CTAO data processing system is data-centric, we refer to the subsystems of the data processing chain as pipelines through which data flows while being transformed (reduced or augmented).

The CTAO operates with three categories of data. Category-A data are the result of the online data analysis and are used to provide science alerts to the astrophysics community and for optimal time-observation schedules. No offline-calculated or refined calibrations are applied to the Category-A data. The Category-B data are generated the day after the observation. They include preliminary offline calibrations, augmenting their quality and precision. This data is released to the PI of the corresponding observation proposal for early analysis. Finally, the Category-C data are produced within one month of the data taking and include state-of-the-art knowledge of the array telescopes' performance and observation conditions. They include, when needed, a set of tailored simulations to improve the qualitative description of the instrument response to the gamma-ray signals. The Calibration and Data Quality pipelines deal with the Category-B and Category-C data.

The Calibration pipeline functionality is divided into three major blocks. The first one concerns atmospheric calibrations. A proper understanding of the atmosphere composition is vital for ground-based gamma-ray astrophysics because the atmosphere plays the role of the radiative medium where the atmospheric showers produce the Cherenkov light. It thus becomes de-facto a part of the detector. The Calibration pipeline provides a comprehensive assessment of the state of the atmosphere, including long-term modelling, as well as the corrections for the short-term transient effects caused by the aerosols and dust particles. The second calibration functionality block addresses the individual telescope calibrations of pointing, timing and gain calibration of the telescope's main camera; and optics calibration, including the optical point spread function and absolute optical throughput. The third functional block performs the inter- and cross-calibration of the whole array aiming at improving the absolute energy scale determination and time calibration of the array.

The Data Quality pipeline delivers the quality metrics and ranks recorded data according to its quality and suitability for certain physics analyses. It consists of frontend and backend packages. The frontend package manages the interfaces with other components of the CTAO Data Processing software and provides a user interface for the operators and scientists of the CTAO. The backend package contains the core algorithms for data quality monitoring and data certification and classification.

The contribution to the development of the Calibration and Data Quality pipelines is led by the DPNC/UniGe group. It covers completely the design and implementation of the Calibration software and foresees a significant contribution to the Data Quality package. On top of this, Mykhailo Dalchenko, coordinator of the Calib Pipeline, is seconded to CTAO at the 50% level as the Data Processing and Preservation System (DPPS) Software Architect and Georgios Voutsinas will be seconded as Calibration Scientist, to determine the scientific impact of the Calibration Pipeline.