Incorrect values in the WEIGHT_SPECTRUM column of interferometric COBALT data

This is an error associated with the COBALT correlator that affects data recorded between 19-03-2014 and 31-10-2014. A solution to this problem is described below. The error has been corrected at the correlator level so all data recorded after 31-10-2014 is not affected by this problem

 

An error occurred in the code of the COBALT correlator which resulted in incorrect values being written to the WEIGHT_SPECTRUM column of interferometric COBALT data. This error arose whenever a station was flagged for either part of, or for the entire observation. COBALT incorrectly wrote -1 in the WEIGHT_SPECTRUM column of the Measurement Set (MS), which eventually results in a weight of around 21.447 (it should be between 0 and 1)

 

Due to the wrong weights, baselines with flagged stations are not actually flagged later on. Their contribution prevents the solver from converging, and can lead to non-optimal demixing and BBS results. This means that there is a chance that some A-team signal is left in the visibilities, so we ask that you inspect your data carefully.

 

The issue was discovered by the Radio Observatory (hereafter RO) while analyzing suspicious demixing results and investigating the reason of long processing time for RO pipelines. Once identified, the problem has been promptly fixed at the correlator level. In particular, all observations performed after 31-10-2014  do not suffer from this issue

Depending on the data type, the issue can be addressed as follows:

 

Raw data: in order to solve the issue the data need to be processed using a new version of the software, no older than 2014-10-30. To check which version of LOFAR software is installed in your own computing facility, run the command:

 

use <Your_Lofar_enviromnent>

versionlofarstman | grep "package revision"

returns (for example): package revision  = 30484 (last change in package)

 

The install is ok if package revision >= 30324.

Processing the data with the new version of the software will automatically solve the problem (with a fix in the LofarStMan storage manager).

If needed, ask your software or system administration to update your LOFAR software. The latest version is always available at:  https://svn.astron.nl/LOFAR/trunk

Build instructions are available at:  http://www.lofar.org/operations/doku.php?id=public:user_software:documentation:lofar-cmake

 

Processed data:

 - If autoweight=TRUE was not used in the pre-processing:

 

If the data was not averaged, flag the data for which WEIGHT_SPECTRUM > 1, for example by using the following TaQL command:

taql ‘update <your_MS> set FLAG=True where any(WEIGHT_SPECTRUM>1) and ANTENNA1!=ANTENNA2

 

If the data was averaged, flag the data for which WEIGHT_SPECTRUM > 1, for example by using the following TaQL command:

taql ‘update <your_MS> set FLAG=True where any(WEIGHT_SPECTRUM > <averaging_frequency_step_used> * <averaging_time_step_used>) and ANTENNA1!=ANTENNA2


- If autoweight=TRUE was used in the pre-processing (default setting in the RO pre-processing pipelines), it is advised to flag data for which there are outliers in the WEIGHT_SPECTRUM column of the MS.

In case you experience issues when running these commands or you need help in any of these steps, please do not hesitate to contact Science Operations and Support through the RO helpdesk which is hosted at https://support.astron.nl/rohelpdesk

Design: Kuenst.    Development: Dripl.    © 2020 ASTRON