CYBERSKA

 A Cyberinfrastructure platform to meet the needs of data intensive radio astronomy on route to the SKA

GMRT Deep Polarization Field

GMRT Deep Polarization Field

GMRT Deep Polarization Field

  • Public

RFI flagging comparison between RFLAG , TFCROP in CASA and the LOFAR flagger for 610 MHz ELAIS N1 deep field data

Last updated 2296 days ago by Preshanth Jagannathan Comments (1)

In CASA 3.4 the old flagdata and flagdata2 tools have been replaced by the tool tflagdata. Tflagdata offers two different flagging algorithms namely tfcrop and rflag. Tfcrop is a thresholding alogrithm while Rflag is an implementation of the algorithm of the same name in AIPS. The two together make a rather robust combination supplementing each other.

The LOFAR flagger as published in Offringa et al is a thresholding based algorithm for automated flagging of LOFAR data. A comparison of the CASA routines with the the LOFAR flagger is the scope of this article.

The data in consideration is from GMRT is a full stokes observation of the ELAIS N1 Deep field pointing 01 from 11 Feb 2012. The observation had 3C286 as the Flux and Polarization calibrator. 1549+506 as the phase calibrator for the course of the observations.

To flag the data of 3C286 effectively an initial bandpass calibration is performed on the uncalibrated measurement set and the flux of 3C286 applied. The data is then split into measurement sets corresponding to each of the calibrators and the target field ready for flagging.

Flagging with TFCrop in CASA 3.4

To flag the data the following parameters were set in CASA and run.

#  tflagdata ::  All-purpose flagging task based on data-selections and flagging modes/algorithms
vis                 = '11feb2012.1549.ms/' #  Name of MS file to flag
mode                =   'tfcrop'        #  Flagging mode (list/manual/clip/shadow/quack/elevation/tfcrop/rflag/extend/unflag/summary)

     field          =         ''        #  Field names or field index numbers: '' ==> all, field='0~2,3C286'
     spw            =         ''        #  Spectral-window/frequency/channel: '' ==> all, spw='0:17~19'
     antenna        =         ''        #  Antenna/baselines: '' ==> all, antenna ='3,VA04'
     timerange      =         ''        #  Time range: '' ==> all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Correlation: '' ==> all, correlation='XX,YY'
     scan           =         ''        #  Scan numbers: '' ==> all
     intent         =         ''        #  Observation intent: '' ==> all, intent='CAL*POINT*'
     array          =         ''        #  (Sub)array numbers: '' ==> all
     uvrange        =         ''        #  UV range: '' ==> all; uvrange ='0~100klambda', default units=meters
     observation    =         ''        #  Observation ID: '' ==> all
     feed           =         ''        #  Multi-feed numbers: Not yet implemented

     ntime          =     'scan'        #  Time-range to use for each chunk (in seconds or minutes)
     combinescans   =      False        #  Accumulate data across scans.
     datacolumn     =     'data'        #  Data column on which to operate (data,corrected,model,residual)

     timecutoff     =        3.0        #  Flagging thresholds in units of deviation from the fit
     freqcutoff     =        3.0        #  Flagging thresholds in units of deviation from the fit
     timefit        =     'poly'        #  Fitting function for the time direction (poly/line)
     freqfit        =     'poly'        #  Fitting function for the frequency direction (poly/line)
     maxnpieces     =          4        #  Number of pieces in the polynomial-fits (for 'freqfit' or 'timefit' = 'poly')
     flagdimension  = 'freqtime'        #  Dimensions along which to calculate fits (freq/time/freqtime/timefreq)
     usewindowstats =     'none'        #  Calculate additional flags using sliding window statistics (none,sum,std,both)
     halfwin        =          1        #  Half-width of sliding window to use with 'usewindowstats' (1,2,3).

action              =    'apply'        #  Action to perform in MS and/or in inpfile (none/apply/calculate)
     display        =   'report'        #  Display data and/or end-of-MS reports at runtime (data/report/both).
     flagbackup     =       True        #  Back up the state of flags before the run

savepars            =      False        #  Save the current parameters to the FLAG_CMD table or to a file
async               =      False        #  If true the taskname must be started using tflagdata(...)
TFCrop is based on a thresholding algorithm designed by Urvashi Rau at the NRAO more information can be found here. The above parameters were found to do the optimal amount of flagging for the phase calibrator and target field data. On Urvashi's suggestion to change flagdimension = 'timefreq' so that time domain is flagged first, lead to no significant change in the data flagged.
The pre-flagged baseline

image


The Baseline flagged using TFCrop in CASA

image


The Baseline flagged using TFCrop(in pink) and Lofar Flagger (in yellow)

image

Flagging with RFLAG in CASA 3.4

#  tflagdata ::  All-purpose flagging task based on data-selections and flagging modes/algorithms
vis                 = '11feb2012.1549.ms/' #  Name of MS file to flag
mode                =    'rflag'        #  Flagging mode (list/manual/clip/shadow/quack/elevation/tfcrop/rflag/extend/unflag/summary)


     field          =         ''        #  Field names or field index numbers: '' ==> all, field='0~2,3C286'
     spw            =         ''        #  Spectral-window/frequency/channel: '' ==> all, spw='0:17~19'
     antenna        =         ''        #  Antenna/baselines: '' ==> all, antenna ='3,VA04'
     timerange      =         ''        #  Time range: '' ==> all,timerange='09:14:0~09:54:0'
     correlation    =         ''        #  Correlation: '' ==> all, correlation='XX,YY'
     scan           =         ''        #  Scan numbers: '' ==> all
     intent         =         ''        #  Observation intent: '' ==> all, intent='CAL*POINT*'
     array          =         ''        #  (Sub)array numbers: '' ==> all
     uvrange        =         ''        #  UV range: '' ==> all; uvrange ='0~100klambda', default units=meters
     observation    =         ''        #  Observation ID: '' ==> all
     feed           =         ''        #  Multi-feed numbers: Not yet implemented

     ntime          =     '2min'        #  Time-range to use for each chunk (in seconds or minutes)


     combinescans   =      False        #  Accumulate data across scans.
     datacolumn     =     'data'        #  Data column on which to operate (data,corrected,model,residual)

     winsize        =          3        #  Number of timesteps in the sliding time window [aips:fparm(1)]
     timedev        =         ''        #  Time-series noise estimate [aips:noise]
     freqdev        =         ''        #  Spectral noise estimate [aips:scutoff]
     timedevscale   =        3.5        #  Threshold scaling for timedev [aips:fparm(9)]
     freqdevscale   =        3.5        #  Threshold scaling for freqdev [aips:fparm(10)]
     spectralmax    =  1000000.0        #  Flag whole spectrum if freqdev is greater than spectralmax [aips:fparm(6)]
     spectralmin    =        0.0        #  Flag whole spectrum if freqdev is less than spectralmin [aips:fparm(5)]

action              =    'apply'        #  Action to perform in MS and/or in inpfile (none/apply/calculate)
     display        =   'report'        #  Display data and/or end-of-MS reports at runtime (data/report/both).
     flagbackup     =       True        #  Back up the state of flags before the run

savepars            =      False        #  Save the current parameters to the FLAG_CMD table or to a file
async               =      False        #  If true the taskname must be started using tflagdata(...)

 

The RFlag algorithm was originally developed by Eric Greisen in AIPS, additional information can be found here. There is clearly defined documentation available for the tasks in the link shared which would enable us to determine the parameter values appropriate for the flagging tasks on hand.

The following parameters were found to provide a good level of broad band RFI flagging which is one place where TFCrop was falling short. The combined use of both RFLAG and RFCrop led to significantly improved results.

Baseline Flagged using TFCrop+RFLAG

image

Baseline Flagged using TFCrop+RFLAG(in pink) and Lofar Flagger (in yellow)

image

Comments

  • Russ Taylor 2294 days ago

    Hi Preshant. Can you provide a plot of the lofAr flagged by itself? It's hard to tell from your second figure if it also flagged the points found by tfcrop.