Wrong information in antenna tables of LOFAR Measurement Sets

During a software roll out a year ago, on 13th February 2013, a bug was inadvertently introduced in a procedure used to store information in the LOFAR internal database (LOFAR_4 OTDB). This bug was related to a function called ‘getbrokenhardware', used by the system to store in the Measurement Sets (MS's) information about malfunctioning antenna elements. It initiated a cumulative effect whereby the information stored in the MS's after the aforementioned date consisted of the addition over time of all newly diagnosed malfunctioning elements to the original broken elements information, while not removing from the list elements which subsequently became operational after repair.

 


The issue was discovered by the EoR KSP while performing deep wide-field calibration and imaging. It affects all reduction software that uses the antenna element information in the MS's to simulate the beam response of the LOFAR array.

Please be aware that the visibilities recorded in your MS's are correct; only the information stored in the Antenna Table is wrong.

Therefore, the quality of the data is still preserved.

Depending on the processing procedures you applied to achieve your final results, these might be affected in different fundamental ways.

In particular, please note the following.

This issue does not affect:

  • Products of Radio Observatory pre-processing, calibrator, and target pipelines;
  • Solution transfer to the target field from calibrators observed with dedicated bandwidth;
  • Calibration results at the field center obtained adopting an external model of the sky, unless your science case requires thermal noise limited maps where the positions and fluxes of off-axis sources become important.

 


This issue affects only interferometric observations that have been processed using:

  • Products of the Radio Observatory imaging pipeline;
  • The ‘smart demix' (currently not available in the Standard Imaging Pipeline);
  • BBS (used for wide-field self calibration);
  • AWImager.


Since the specific effects depend on the state of the list on the observing date, as well as on the characteristics of the observed fields, it is hard to quantify for all interferometric observations performed during the last year.

Luckily it is possible to extract a history of broken and repaired antenna elements since the moment that the bug was introduced into the system.

This problem was solved during a software roll out on 10th February 2014. From that date onward all MS's antenna tables report the correct information again.

 

The script to correct for this issue can be downloaded from the link below:

 

fixinfo.tar


We advise Users to compare the results obtained after correcting the MS's antenna tables with those with wrong information in order to estimate the impact of this issue on their scientific goals.

The software requirements to run this script on your systems are:

 

1. TaQL (so casacore)

2. pyrap

3. LofIm (in particular, it uses makebeamtables and addfailedtileinfo)

 

Usage:

tar xvf fixinfo.tar

cd fixinfo

#running the program with full path is needed at some systems, you can use `pwd` to get the path name #on most systems.

full/path/to/fixinfo/fixbeaminfo path/to/my_ms/my_ms.MS

 

The updated number of flagged elements can be listed, e.g.,

 

taql 'select ANTENNA_ID, ntrue(ELEMENT_FLAG), [select NAME from my_ms.MS::ANTENNA][ANTENNA_ID]  from my_ms.MS::LOFAR_ANTENNA_FIELD'

 

This will give something like:

 

0 48 CS001HBA0

1 48 CS001HBA1

2 48 CS002HBA0

3 56 CS002HBA1

...

55 0 RS407HBA

56 0 RS409HBA

57 6 RS503HBA

58 0 RS508HBA

 

Note that in this case, it concerns the HBA dual mode, so half the tiles (24=48 polarizations) are flagged for HBA0 etc. CS002HBA1 in the above example, has HBA0 flagged (48 pols) + 4 additional tiles (8 pols) because they were set to broken.

The plots of the history of operational antenna elements for all LOFAR stations since February 2013 can be found here.

In case you experience issues when running this script, please contact Science Operations and Support through the RO helpdesk which is hosted at https://support.astron.nl/rohelpdesk 

 

 

 

 

Design: Kuenst.    Development: Dripl.    © 2020 ASTRON