Pre-processing Pipeline


    Processing of the raw uv data, which consists of calibration and imaging steps, is handled offline via a series of automated pipelines (as detailed here).

    The first standard  data processing step performed by ASTRON is described below and is called the Pre-processing Pipeline: 

    Pre-Processing Pipeline: Flags the data in time and frequency, and optionally averages them in time, frequency, or both (the software that performs this step is labeled DPPP - Default Pre-Processing Pipeline). This stage of the processing also includes, if requested, a subtraction of the contributions of the brightest sources in the sky (the so called "A-team": Cygnus A, Cassiopeia A, Virgo A, etc...) from the visibilities through the 'demixing' algorithm (B. van der Tol, PhD thesis).  Currently, users should specify if demixing is to be used, and which sources should be demixed. Visibility averaging should be chosen to a level that reduces the data volume to a manageable level, while minimizing the effects of time and bandwidth smearing. The averaging parameters, as well as the estimated storage capacity required in the LOFAR Long Term Archive, are also specified by the users through the North Star proposal submission tool.  

    Users willing to further process data products generated from the RO pre-processing pipeline can use advanced calibration/imaging pipelines. Currently these pipelines are not offered in production and for Cycle 13 proposals CEP3 processing time can be requested by users to perform advanced-calibration in a standalone fashion, as described in the LOFAR Imaging Cookbook.  If you are willing to adopt this CEP3 offline option please make this clear in your proposal by answering the relevant question in the technical section of the proposal "Off-line data processing on RO facilities (CEP3) requirement". Alternatively, proposers may download, install and run available advanced pipleines in their own computing facilities.

     

    Access to Data Products    

    All final products will be stored at the LOFAR Long Term Archive where, in the future, significant computing facilities may become available for further re-processing. Users can retrieve datasets from the LTA for reduction and analysis on their own computing resources or through the use of suitable resources on the GRID. More information are available here.

    From the moment the data are made available to the users at the LTA the users will have four weeks available to check the quality of their data and report problems to the Observatory. After this time window has passed, no requests for re-observation will be considered.

     

Design: Kuenst.    Development: Dripl.    © 2020 ASTRON