Scope

History

Version 1.0  - 21 Feburary 2002. Chris Tinney
Version 1.1  - 21 January 2005. Chris Tinney (Updated to provide advice on use of CASU WFI toolkit)

General Guidelines

Environments

Your data can be reduced in

• IRAF (using the mscred package)
• or in the PIPELINE gui front end to IRAF mscred,
  
• or using the Cambridge Astronomical Surveys Unit imaging imaging data reduction toolkit  (as used on the INT Wide Field Survey)
  

0. BACKGROUND & CURRENT STATUS

1. CASU IMAGING TOOLKIT

2. PIPELINE / IRAF

The IRAF MSCRED-based data reduction pipeline GUI is started on aaowfi with the command

pipeline -cicada &

It is basically a GUI front-end to the IRAF ccdproc tasks running under the IRAF mscred package. It is recommended you read the IRAF mscred documentation in order to understand what pipeline is doing to your data.

In essence however, pipeline will overscan subtract, trim, zero subtract, dark subtract and flat field your data, if you have a Zero frame a Dark frame and a set of useful flat fields already created. At present you have to use IRAF directly to create these calibration files.

Bias Subtraction

Bias levels fluctuate up and down by several adu on all timescales, so overscan subtraction is essential. Determining a constant value for each CCD in each image from the overscan regions, and subtracting it.

The bias frames show some low-level structure, so subtraction of a bias frame constructed from 7-15 overscan-subtracted bias frames, from each image is recommended.

Dark Subtraction

WFI is operated quite warm, so dark subtraction is essential. You should a dark frame from  overscan and bias subtracted dark data, and subtracted it (scaled for exposure time) from every image.

Flat Fielding

Domeflats work well for i and z. Twilight flats work resonably well for all passbands.
 
 

Details - Overscan and Bias subtraction

Overscan : You can overscan subtract your data  using the regions defined in the BIASSEC and TRIMSEC FITS keywords (this is what pipeline will do, and it is the default action of IRAF's mscred ccdproc unless you specifiy otherwise). If these keywords are not present in your data, then you can add them in IRAF as follows.
 

hedit *.fits[1] BIASSEC '[2075:2098,5:4098]' add=yes verify=no
hedit *.fits[2] BIASSEC '[2075:2098,5:4098]' add=yes verify=no
hedit *.fits[3] BIASSEC '[2075:2098,5:4098]' add=yes verify=no
hedit *.fits[4] BIASSEC '[2075:2098,5:4098]' add=yes verify=no
hedit *.fits[5] BIASSEC '[1:24,39:4132]' add=yes verify=no
hedit *.fits[6] BIASSEC '[1:24,39:4132]' add=yes verify=no
hedit *.fits[7] BIASSEC '[1:24,39:4132]' add=yes verify=no
hedit *.fits[8] BIASSEC '[1:24,39:4132]' add=yes verify=no
hedit *.fits[1] TRIMSEC '[16:2059,5:4098]' add=yes verify=no
hedit *.fits[2] TRIMSEC '[16:2059,5:4098]' add=yes verify=no
hedit *.fits[3] TRIMSEC '[16:2059,5:4098]' add=yes verify=no
hedit *.fits[4] TRIMSEC '[16:2059,5:4098]' add=yes verify=no
hedit *.fits[5] TRIMSEC '[41:2084,39:4132]' add=yes verify=no
hedit *.fits[6] TRIMSEC '[41:2084,39:4132]' add=yes verify=no
hedit *.fits[7] TRIMSEC '[41:2084,39:4132]' add=yes verify=no
hedit *.fits[8] TRIMSEC '[41:2084,39:4132]' add=yes verify=no

The recomended parameters for the iraf ccdproc overscan subtraction (and trim) are
 

ccdproc images=@filein.lis output=@fileout.lis noproc=no ccdtype='' xtalkco=no oversca=yes trim=yes zerocor=no darkcor=no flatcor=no function=legendre order=1 sample='*' naverage=1 niterate=2 low_rej=3 high_rej=3

Bias (or zero) subtraction : Create a zero correction using at least 7 bias frames from your run, after overscan subtraction and binning.

zerocombine input=@zero.lis output=Zero combine=median reject=sigclip lsigma=6 hsigma=3 mclip=yes scale=none

Apply the zero correction to all images

ccdproc images=@filein.lis output=@fileout.lis noproc=no ccdtype='' xtalkco=no oversca=yes trim=yes zerocor=yes darkcor=no flatcor=no zero=Zero function=legendre order=1 sample='*' naverage=1 niterate=2 low_rej=3 high_rej=3

Details - Linearity

Unfortunately, IRAF does not offer linearity correction in its usual CCD processing. You can insert the derived linearity corrections into the file headers of each image extension as follows, and then apply them, but the process is quite inefficient.
 

hedit *.fits[1] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[1] LIN_A0 1.0      add=yes update=yes
hedit *.fits[1] LIN_A1 0.0      add=yes update=yes
hedit *.fits[1] LIN_A2 3.86e-12 add=yes update=yes
hedit *.fits[1] LIN_A3 0.0      add=yes update=yes

hedit *.fits[2] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[2] LIN_A0 1.0      add=yes update=yes
hedit *.fits[2] LIN_A1 0.0      add=yes update=yes
hedit *.fits[2] LIN_A2 3.18e-12 add=yes update=yes
hedit *.fits[2] LIN_A3 0.0      add=yes update=yes

hedit *.fits[3] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[3] LIN_A0 1.0      add=yes update=yes
hedit *.fits[3] LIN_A1 0.0      add=yes update=yes
hedit *.fits[3] LIN_A2 8.43e-12 add=yes update=yes
hedit *.fits[3] LIN_A3 0.0      add=yes update=yes

hedit *.fits[4] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[4] LIN_A0 1.02753     add=yes update=yes
hedit *.fits[4] LIN_A1 -1.6555e-6  add=yes update=yes
hedit *.fits[4] LIN_A2 1.7791e-11  add=yes update=yes
hedit *.fits[4] LIN_A3 0.0         add=yes update=yes

hedit *.fits[5] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[5] LIN_A0 1.0      add=yes update=yes
hedit *.fits[5] LIN_A1 0.0      add=yes update=yes
hedit *.fits[5] LIN_A2 4.26e-12 add=yes update=yes
hedit *.fits[5] LIN_A3 0.0      add=yes update=yes

hedit *.fits[6] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[6] LIN_A0 1.0      add=yes update=yes
hedit *.fits[6] LIN_A1 0.0      add=yes update=yes
hedit *.fits[6] LIN_A2 5.93e-12 add=yes update=yes
hedit *.fits[6] LIN_A3 0.0      add=yes update=yes

hedit *.fits[7] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[7] LIN_A0 1.0      add=yes update=yes
hedit *.fits[7] LIN_A1 0.0      add=yes update=yes
hedit *.fits[7] LIN_A2 -5.50957e-12 add=yes update=yes
hedit *.fits[7] LIN_A3 2.39334e-16  add=yes update=yes

hedit *.fits[8] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[8] LIN_A0 1.0      add=yes update=yes
hedit *.fits[8] LIN_A1 0.0      add=yes update=yes
hedit *.fits[8] LIN_A2 5.37e-12 add=yes update=yes
hedit *.fits[8] LIN_A3 0.0      add=yes update=yes

IRAF does not have a way to include linearity correction in its ccdproc processing. Nor is there a really useful linearisation routine. You have to use imexpr which is not bright enough to accept a list of input/output images. Furthermore, you can't make msccmd and imexpr interact in the way they should, so you have to split each MEF image into individual CCDs, then correct each individually, then join the images back together.

If an IRAF guru can provide a script/executable to speed this process up that would be incredibly useful. All advice gratefully received.

In principle, it should be straight-forward to put this a useful IRAF script. However, the IRAF scripting language is such an abortion I couldn't be bothered, so instead here is a recipe to make a giant list of commands which you can pipe into CL.

#
# Some csh to make a long file to linearise data in the directory
# /data1/cgt/FEB01/20010202/A/linearity/zeroed
# and put the results into the directory directly above.
#
echo mscred > doit
echo cd /data1/cgt/FEB01/20010202/A/linearity/zeroed >> doit
foreach j ( 02feb*.fits )
   echo mscsplit input=$j mefext=.$j:e delete=no verbose=yes >> doit
   foreach i ( 1 2 3 4 5 6 7 8 )
      echo "imexpr 'a*(a.lin_a0 + a.lin_a1*a + a.lin_a2*a**2 + a.lin_a3*a**3)' \
      output=l${j:r}_$i a=${j:r}_$i" intype=double outtype=real refim=a >> doit
   end
   echo imrename ${j:r}_0 l${j:r}_0 >> doit
   echo mscjoin input=l${j:r} output=../${j:r} delete=yes verbose=yes >> doit
   echo imdel ${j:r}_\*.fits go_ahead=yes verify=no default=yes >> doit
end
echo logout >> doit

cl < /data1/cgt/FEB01/20010202/A/linearity/zeroed/doit

Note that the imexpr processing is incredibly slow. However it does work. A dedicated routine which operated directly on MEF files would be much faster. It could also be optimised to only calculate terms for which the coefficient is non-zero.
 
 

Details - Dark subtraction

Details - Flat Fielding

IRAF MSCRED Processing Hints

Changing FITS keywords in your files. The trick is to know where the keywords are. Some keywords apply to the entire file (eg. FILTER). In MSCRED parlance they sit in the [0] extension. Other keywords apply to specific images (eg. TRIMSEC) and they sit in the [1],[2], ... extensions.

The tricky bit is if you try the command

hedit 02feb0166[1] FILTER "z #98 (WFI SDSS)"  upd+

in an attempt to update the filter name, MSCRED will TELL you it has changed it. But it won't have. You need to use

hedit 02feb0166[0] FILTER "z #98 (WFI SDSS)"  upd+

This one did frustrate me for days. I could NOT work out why I couldn't update a header entry. Note that

ccdhedit 02feb0166 FILTER "z #98 (WFI SDSS)"

will also not work.

How to linearise your data? See above.

How to put correct BIASSEC and TRIMSEC entries in your file header? See above.

Can't get your entire image to display in a 1 CCD pixel to 1 display pixel format? Edit your login.cl file and change the 'set stdimage' line to

       set stdimage = imt8800

but be aware your images will take somewhat longer to load!