AAO Wide Field Imager & Prime Focus Unit


Reduction Guide

 
 
 

Version 1.1 - 21 January 2005
 
 

Chris Tinney


Useful WFI links


 

Scope

This document is a guide to what we've learned about reducing WFI data.


History

Version 1.0  - 21 Feburary 2002. Chris Tinney
Version 1.1  - 21 January 2005. Chris Tinney (Updated to provide advice on use of CASU WFI toolkit)



General Guidelines

Environments

Your data can be reduced in

0. BACKGROUND & CURRENT STATUS

As part of its planning for the implementation of the Wide Field Imager (WFI), AAO/RSAA/U.Melbourne had PLANNED to implement a data processing pipeline for the reduction of WFI data. Three (then) powerful Ultra SPARC computers to handle the significant load imposed by WFI reductions were funded from a LIEF program at AAO, RSAA and U.Melbourne. A GUI front-end to streamline the use of the IRAF mscred tasks was also implemented by RSAA. (It is worth noting that the WFI partners did not have the resources to develop a new WFI data processing system from scratch - we were always going to have to rely on “standing on the shoulders”  of others.)

Several years down the track now, the reduction of WFI data remains problematic. The Ultra SPARCs acquired to do such data reduction (especially when running IRAF) were always fairly slow, and the mscred package was neither particularly powerful, nor flexible enough to handle the complexities of processing WFI data (eg. it was unable to do linearity corrections in any straightforward way).

However, the remorseless march of computer power has given us some advantages – the 3.3 GHz HP Linux workstation acquired for data reduction at the AAT (a copy was also purchased for Epping) is so much faster for WFI data reduction that the AAT WFI Ultra SPARC “stands still” by comparison.

Similarly, the advent of other mosaics on other telescopes has meant that other observatories have had to implement data processing systems. Unfortunately, it has to be said that no other observatory we are aware of has produced a system which is packaged and documented to the extent that it can be easily implemented by the AAO for the routine reduction of all WFI data at the telescope. This is in contrast to infrared data reduction, where the ORAC-DR system has been able to be implemented by the AAO and delivers excellent on-line reductions to visiting astronomers.


1. CASU IMAGING TOOLKIT

Among the facilities to produce a mosaic processing system is the Cambridge Astronomical Surveys Unit (CASU) who have extended their APM system, to data from a variety of mosaic imagers. This system has been used by CASU to process data from a variety of telescopes (ESO/MPI 2.3m [at ESO’s invitation], CFHT12K, KPNO mosaic, INT), including the WFI on the AAT.

However, it must be emphasised that the CASU processing system is not a “pipeline”, rather it is a “toolkit”. It will not, when pointed at a directory of data, faultlessly and seamlessly reduce it for you. Its not designed for that. What it will do is give you the tools to process the data in about one (long) afternoon for a single band from a night’s observing. But ….

•    It requires the application of an understanding of data processing (it's not “idiot proof” or automatic).
•    It requires interacting with it at a command-line level.
•    It is not well documented.
•    It is not supported.

The last two points are particularly important to bear in mind. CASU is not funded to, nor do they have the resources to, support their toolkit as a general user facility. They are prepared to let the AAO use it, and they can answer some (small) number of specific questions about it. But they can’t support it as an ‘IRAF-like’ or ‘Starlink-like’ data processing system.

Bearing all this in mind then, the status of the CASU/APM Toolkit at the AAO is as follows

•    On an observing run at the AAT last year, Simon Hodgkin (CASU) installed a (then current) version of the toolkit at the AAT. This installation was not competed to a fully operational status, but was sufficient to characterise the WCS for WFI on the AAT.
•    Rob Sharp (2dF Fellow, formerly at Cambridge) has had some considerable experience in processing mosaic data with the CASU Toolkit at Cambridge. He has installed a version of the toolkit on our data reduction Linux machine (supplemented by some libraries from the older AAT installation). This version of the software has (as part of Rob’s research activities) been "bodged to reduce" WFI data.  “Bodged” because while implemented and running, the toolkit is not seamless in processing AAT data – it has “features”, mostly having to do with WFI header items being either non-standard or otherwise different form what the toolkit expects.
•    However, following a set of notes provided by Rob Sharp (see below), others have been able to reproduce a reduction of a set of observations (of the cluster AC118) at 9 WFI pointings, resulting in a final mosaic image 25000x25000 pixels in size. The processing is straightforward, and it is clear that a pass-band’s worth of one night's WFI data can be processed in about one solid day of work on the HP Linux workstation.

The result of which is that the AAO can support the reduction of WFI data at Epping, using our workstations and the CASO toolkit. It is not slick, and its not elegant, and we can’t support the reduction of data elsewhere, since installation of the toolkit is non-trivial and it’s the property of CASU. The AAO’s in-house resources to support the streamlining of the CASU toolkit for AAT data are limited. We are making an effort to address some of the points listed at the head of the attached notes below, but they can only be carried out on a “best efforts” basis. Also we currently only have on computer on which the software will process WFI data efficiently. And only one user can use this computer (and its disk space) at a time. So any visitors wishing to use the system will need to ‘book’ it in advance for a fixed period of time. Please contact Scott Croom or Chris Tinney to do this.

Finally, if any WFI users are aware of a ‘turnkey’ mosaic data reduction system that we can import from elsewhere, please let us know!

CASU Toolkit WFI Reductions – A sample README file

WFI Toolkit issues to address (CGT)

o tools for 'preprocessing' AAT data to fix up headers to FITS standard
  (eg. axis information, extension cleanup, EXPTIME, etc)
o Fix AAT filter information to what Mike's system expects
o Make it work without need for “perl /home/aaossz/rgs/……
o Make it get to Angle.pm, CTITSIO etc properly. Ie. get the paths and perl installation sorted out.
o How to implement linearity corrections (necessary for AAT data)
o Scattered light / light concentration – is it present in AAT data? How to measure/correct.
o WFI bad pixel masks – create a default set. Also how to flag saturated pixels?


===========================
Some note on Rob Sharp’s experience playing with the APM wide field imaging pipeline (and a few CGT notes)

The pipeline currently available at Epping on aaolxa, is
one installed by Rob Sharp (that he had installed at Cambridge
and so is about 6 months old), supplemented by some routines
he didn't have, so copied from the Site installation done by
Simon Hodgkin about 12 months ago.

On aaolxa,

cd /data2_lxa/cgt/WFI
   This directory contains both raw data I gave to Rob Sharp for him to
   play with, and some results from processing he did on AC118

   eg. cd AC118

   This directory contains the results of processing images 09aug0081-88 -
   the result is a 1.3Gb image mosaic.fits!


Rob Sharp has pointed me to a few 'help' files

To look at the data
   To display these images one can use (on my Mac powerbook)
      gaia (works even for 1.3Gb images!)
      jsky (works for 200Mb images but not 1.3Gb images),
      ds9 (works for 200Mb images but not 1.3Gb images)

      fits2ndf will convert 200Mb images to figaro format, but not 1.3Gb images.

      figaro/figdisp will not display 200Mb images
       "Display resolution is too large for this
        routine to function properly."
        -> probably a tvpckg error.


================================================================================
/data2_lxa/cgt/WFI/processing.txt (Rob's processing notes)

# First source the CASU software

source /home/aaossz/rgs/cfh12k/casu_setup

## 20010809
# Now move the junk data
# This gets rid of all the single CCD frames and text files and stuff
cd /data2_lxa/rgs/wfi/20010809/A
perl /home/aaossz/rgs/cfh12k/processing/mvifsize.pl 138893760

# Now try and see what the data is, this will no-doubt fail due
# to missing header keywords.  Such is life.
perl /home/aaossz/rgs/cfh12k/processing/what.pl "*.fits" > list

# Run the header checker
perl /home/aaossz/rgs/cfh12k/processing/checkheader.pl

# All seems well (but it wasn't, as I found out later).
# Try preprocessing the data, ie. split it up based on filter etc...
perl /home/aaossz/rgs/cfh12k/processing/preprocess.pl

# This makes a series of new directories and moves the data.
# Most files get moved into new directorys (Bband, iband, bias, dark)
# The ones that are left are the ones that have obvious header defects
#
# There was an additional problem here.  Some of the filters have defective filter keywords.
# They are oftne listed as #NN WFI SDSS filter.  Presumably #NN is the filter ID.
# However, the script is finding the I in WFI and making Iband filter files.
# These tend to be a mixture of U, V and i badn images.
# I'd propose to simply alter the FILTER keyword in the header to be a single characer.
# The best way to do that is to run the WHAT.pl script and then HEDIT the images.

perl /home/aaossz/rgs/cfh12k/processing/what.pl "*.fits" > list2
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "my_image.fits" FILTER U

# The software seems to be struggleing with the z band filter data.
# the problem is that the FILTER keyword is missing from most of the
# data. The image NAME has the filter in it but the FILETR keyword is
# set to NONE.  Going to have to IRAF the data back into shape.
# lets try using the CASU hedit.pl routine first though

perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0050.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0051.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0052.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0053.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0054.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0055.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0056.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0057.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0058.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0059.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0060.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0063.fits" FILTER r
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0064.fits" FILTER r
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0080.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0081.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0082.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0083.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0084.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0085.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0086.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0087.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0088.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0092.fits" FILTER r
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0093.fits" FILTER r
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0094.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0095.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0200.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0201.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0202.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0203.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0204.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0205.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0206.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0207.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0208.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0209.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0210.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0211.fits" FILTER i
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0212.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0213.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0214.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0217.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0219.fits" FILTER z
perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "09aug0220.fits" FILTER r

# Running preprocess.pl again should finish splitting the data NOTE :
# this classifies images based on whats in the headers No level or
# exposure tiem checkign is done and if the headers wrong then so is
# the classification.

perl /home/aaossz/rgs/cfh12k/processing/preprocess.pl

# Try making a bias frame (I'm only using 1 nights data at the moment,
# that will be rathewr noisey)

cd /data2_lxa/rgs/wfi/20010809/A/bias/
ls *.fits >bias.list
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @bias.list bias.fits

# It's Best to check this images out, does it look okay?
# I just use IRAF and check a few extensions

display bias.fits[1] 1

# Okay now lets make the dark frames.  Again I'm just using 1 night
#  here.  Probably want to use a lot more.

cd /data2_lxa/rgs/wfi/20010809/A/dark/
ls *.fits >dark.list
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @dark.list dark.fits

## - NOTE the deliberate mistake here!  I forgot to actually bias subtract the DARKS!!
## Lets try again
cd /data2_lxa/rgs/wfi/20010809/A/dark/
rm dark.list
rm dark.fits
ls *.fits >dark.list
/home/aaossz/rgs/cfh12k/soft/fitsio/fitsio_ccdproc @dark.list bias="../bias/bias.fits"

# You will get asked some questions now.  In general the defaults are
# what you want to use so just hit return a few times.  This one
# doesn't automate well, since you need to answer the questions.  There must be
# some way of pipeling the answers in form a file though.

# Now stack the BIAS-subed DARK frames.
ls p[0-9][0-9][a-z][a-z][a-z][0-9][0-9][0-9][0-9].fits >dark_p.list
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @dark_p.list dark.fits

#examinging these on lapcgt they look OK.

# Dark current scaling : I don't know if the software is clever enough
# to know that the dark current frames need to have the same exposure
# time.  It might try scaling them but I don't know if it does or
# not.

# There does seem to be quiet a bit of structure in the DARK frames.
# There are some hot columns in the BIAS frames as well.  I'm going to
# do a full 2D bias and Dark subtraction.  This probably makes the
# images a little noisier, if we haven't got a truck load of bias and
# dark frames, but it will make things look prettier and remove a lot
# of spurions from the file object catalogues.  Since I normaly work
# with colour space outliers, removing the junk give better results
# than gettine absolute best noise performance.
#
# CGT - and in any case read noise and dark noise are irrelevant compared
#      to sky noise for any useful 4m WFI data!
#

# Now we need to check out the other nights to see if we can find some
# more bias and dark frames.
# We do the same mvifsize.pl, what.pl, preprocess.pl and hedit.pl steps as required.

# Looks like the calibration tape didn't have many extra BIAS and DARK
# frames on it.  For a proper reduction you probably need more bias
# and dark frames.

# Now lets find the flat fields
cd /data2_lxa/rgs/wfi/flat/iband/
perl /home/aaossz/rgs/cfh12k/processing/what.pl "../../200108*/A/Iband/*.fits" > list
perl /home/aaossz/rgs/cfh12k/processing/what.pl "../../200108*/A/iband/*.fits" > list2

# looks like there is a problem with the FIlter splitting in preprocess.pl
# The filters seem to be in as #92 (WFI SDSS) - i or  #48 (WFI Schott) - U
# the preprocess.pl script picks up on the I in WFI!
# This means we have iband data and Iband data where the Iband is just a random subset

# I've moved all of the Iband Twilight and dome flats flats to a new direcotry.
cd /data2_lxa/rgs/wfi/flat/Iband

# Do a quick level check with IRAF
imstat /data2_lxa/rgs/wfi/flat/Iband/*.fits[1]
#               IMAGE      NPIX      MEAN      MODE
 /data2_lxa/rgs/wfi/flat/Iband/10aug0136.fits[1]   8263547    22947.    22208.
 /data2_lxa/rgs/wfi/flat/Iband/11aug0011.fits[1]   8256208    11870.    11530.
 /data2_lxa/rgs/wfi/flat/Iband/11aug0012.fits[1]   8256837    12633.    12254.
 /data2_lxa/rgs/wfi/flat/Iband/12aug0017.fits[1]   8276712    28032.    26969.
 /data2_lxa/rgs/wfi/flat/Iband/12aug0018.fits[1]   8276809    28086.    27023.
 /data2_lxa/rgs/wfi/flat/Iband/12aug0019.fits[1]   8276806    28068.    27006.
 /data2_lxa/rgs/wfi/flat/Iband/12aug0020.fits[1]   8277002    28007.    26946.
 /data2_lxa/rgs/wfi/flat/Iband/12aug0021.fits[1]   8277062    28047.    26990.
 /data2_lxa/rgs/wfi/flat/Iband/13aug0141.fits[1]   8270041    21629.    20724.
 /data2_lxa/rgs/wfi/flat/Iband/13aug0142.fits[1]   8272108    21821.    20896.
 /data2_lxa/rgs/wfi/flat/Iband/13aug0143.fits[1]   8269537    20130.    19307.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0009.fits[1]   8257605    17313.    16540.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0010.fits[1]   8259760    16073.    15388.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0011.fits[1]   8251324    14484.    13864.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0135.fits[1]   8271714    14723.    14061.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0136.fits[1]   8272055    19516.    18646.
 /data2_lxa/rgs/wfi/flat/Iband/14aug0137.fits[1]   8281500    26671.    25403.
 /data2_lxa/rgs/wfi/flat/Iband/15aug0004.fits[1]   8264173    20294.    19814.
 /data2_lxa/rgs/wfi/flat/Iband/15aug0005.fits[1]   8277796    27667.    26551.
 /data2_lxa/rgs/wfi/flat/Iband/15aug0006.fits[1]   8286387    26732.    25440.
 /data2_lxa/rgs/wfi/flat/Iband/16aug0009.fits[1]   8283580    20403.    19447.
 /data2_lxa/rgs/wfi/flat/Iband/16aug0010.fits[1]   8284016    22247.    21195.
 /data2_lxa/rgs/wfi/flat/Iband/16aug0011.fits[1]   8288662    24805.    23596.

# Since all the levels look okay I'll just stack the Twis and domes.
# NOTE we need to process them first!!

# Probably shouldn't have dark corrected the flat frames. Arse.

/home/aaossz/rgs/cfh12k/soft/fitsio/fitsio_ccdproc @I_twi.list bias="../../20010809/A/bias/bias.fits" dark="../../20010809/A/dark/dark.fits"

/home/aaossz/rgs/cfh12k/soft/fitsio/fitsio_ccdproc @I_dome.list bias="../../20010809/A/bias/bias.fits" dark="../../20010809/A/dark/dark.fits"

# Now we stack them
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @I_twi_p.list I_twilight.fits
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @I_dome_p.list I_dome.fits

# Correct the filter screwup If you don't do this then the software
# wont tlet you flatfield the i data with an I flat.

perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "*.fits" FILTER i

# To get a good fringe frame, I'm going to stack all the i data from 20010809.
# Since there isn't much data taken, I probably want to use the i data from the whole run
# But that isn't possible with the current disc space.

# Lets just copy all the AC118 data to a new directory

cd /data2_lxa/rgs/wfi/20010809/A

ls *.fits > bias_dark_flatten.list
/home/aaossz/rgs/cfh12k/soft/fitsio/fitsio_ccdproc @bias_dark_flatten.list bias="../bias/bias.fits" dark="../dark/dark.fits" flat="../../../flat/iband/I_twilight.fits"

#
# Doesn't look too bad - most of the remaining structure could well be fringing.
#

# Now Stack the frames to make a fringe frame.  Normaly you should
# check to make sure that the dithers between each frame are large
# enough to nake them independant (ie no star in the same place).

ls p09aug008?.fits > fringe_i.list
/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_combine @fringe_i.list ifringe.fits

# Defringe the data
perl /home/aaossz/rgs/cfh12k/processing/defringe.pl "p*.fits" ifringe.fits auto

#
# Again - result doesn't look too bad.  CCD4 is about 150/9150 counts fainter than the
# rest, and CCD8 is about 100 counts brighter.
#

# Make some object catalogue and have a pop at the astrometry
rm *cat*
# I had to copy the CASU Angle.pm perl modual into this directory,
# the path isn't finding it correctly on it's own.

perl /home/aaossz/rgs/cfh12k/processing/initwcsimage_aat.pl "pp*.fits"

# At this stage, we should make a confidance map based on the flat
# field levels and known bad pixels I forgot to do this.  There is
# software to make this from two flat fields with different (factor of
# 2-3) illumiantion levels.

# more_astrom.pl is complaiing that the EXPTIME keyword isn't set.
# which it isn't because AAT used EXPOSED keyword instead. So want to
# create EXPTIME. Bodge what.pl to produce a 'clean' output I can use
# to create a set oif hedit scripts to make EXPTIME form the EXPOSED keyword.

perl ./what.pl "*09aug*.fits" |  \
   awk '{print "perl /home/aaossz/rgs/cfh12k/processing/hedit.pl", $1, "EXPTIME", $5, "add"}' > crap
source crap
  
#perl /home/aaossz/rgs/cfh12k/processing/hedit.pl "*09aug008*.fits" EXPTIME 600

perl /home/aaossz/rgs/cfh12k/processing/imcore.pl "pp*.fits" noconf 5 5 1 4
perl /home/aaossz/rgs/cfh12k/processing/lot_astrom.pl
perl /home/aaossz/rgs/cfh12k/processing/list_astrom.pl 0.1
perl /home/aaossz/rgs/cfh12k/processing/more_astrom.pl 0.5
perl /home/aaossz/rgs/cfh12k/processing/more_astrom.pl 0.2
perl /home/aaossz/rgs/cfh12k/processing/list_astrom.pl 0.1
perl /home/aaossz/rgs/cfh12k/processing/updateheader.pl "pp09aug008?.fits"
perl /home/aaossz/rgs/cfh12k/processing/list_astrom.pl 0.05

# In order to get the sacleing corrrect for a mosaic, there are some
# fudge factors to derive

perl /home/aaossz/rgs/cfh12k/processing/lot_classify.pl
ls -1 *008?_cat.fits > clist
/home/aaossz/rgs/cfh12k/soft/fitsio/fitsiotableplot clist
perl /home/aaossz/rgs/cfh12k/processing/lot_percent.pl "pp*008?_cat.fits" 8
perl /home/aaossz/rgs/cfh12k/processing/lot_zeropt.pl "pp*008?_cat.fits"
perl /home/aaossz/rgs/cfh12k/processing/updateheader.pl "pp09aug008?.fits"

# These images have very large (order of the chip size) dithers so we
# can't simply stack each chip.  Lets try making a mosaic.

/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_mosaic pp09aug0080.fits test.fits scrunch=1

/home/aaossz/rgs/cfh12k/soft/fitsio/prototype/fitsio_mosaic @mosaic.list mosaic.fits scrunch=1



#
#
#

# try displaying it
catfits /data2_lxa/rgs/wfi/AC118/mosaic.fits
EXT#  FITSNAME      FILENAME              EXTVE DIMENS       BITPI OBJECT      

0     /data2_lxa/rg /wfi_raw2/cicada/2001       25638x25535  16    AC118-1     

display /data2_lxa/rgs/wfi/AC118/mosaic.fits[1:5000,1:5000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[5001:10000,1:5000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[10001:15000,1:5000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[15001:20000,1:5000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[20001:25000,1:5000] 1

display /data2_lxa/rgs/wfi/AC118/mosaic.fits[1:5000,5001:10000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[5001:10000,5001:10000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[10001:15000,5001:10000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[15001:20000,5001:10000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[20001:25000,5001:10000] 1

display /data2_lxa/rgs/wfi/AC118/mosaic.fits[1:5000,10001:15000] 1 z1=8700. z2=10000 zs- zr-
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[5001:10000,10001:15000] 1 z1=8700. z2=10000 zs- zr-
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[10001:15000,10001:15000] 1 z1=8700. z2=10000 zs- zr-
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[15001:20000,10001:15000] 1 z1=8700. z2=10000 zs- zr-
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[20001:25000,10001:15000] 1 z1=8700. z2=10000 zs- zr-

display /data2_lxa/rgs/wfi/AC118/mosaic.fits[1:5000,15001:20000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[5001:10000,15001:20000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[10001:15000,15001:20000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[15001:20000,15001:20000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[20001:25000,15001:20000] 1

display /data2_lxa/rgs/wfi/AC118/mosaic.fits[1:5000,20001:25000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[5001:10000,20001:25000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[10001:15000,20001:25000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[15001:20000,20001:25000] 1
display /data2_lxa/rgs/wfi/AC118/mosaic.fits[20001:25000,20001:25000] 1


2. PIPELINE / IRAF

The IRAF MSCRED-based data reduction pipeline GUI is started on aaowfi with the command
pipeline -cicada &
It is basically a GUI front-end to the IRAF ccdproc tasks running under the IRAF mscred package. It is recommended you read the IRAF mscred documentation in order to understand what pipeline is doing to your data.

In essence however, pipeline will overscan subtract, trim, zero subtract, dark subtract and flat field your data, if you have a Zero frame a Dark frame and a set of useful flat fields already created. At present you have to use IRAF directly to create these calibration files.

Bias Subtraction

Dark Subtraction

WFI is operated quite warm, so dark subtraction is essential. You should a dark frame from  overscan and bias subtracted dark data, and subtracted it (scaled for exposure time) from every image.

Flat Fielding

Domeflats work well for i and z. Twilight flats work resonably well for all passbands.
 
 

Details - Overscan and Bias subtraction

Fitting a constant to the overscan region and subtracting is recommended, followed by the subtraction of a zero/bias frame made up from 9-15 biases taken during your run.

Overscan : You can overscan subtract your data  using the regions defined in the BIASSEC and TRIMSEC FITS keywords (this is what pipeline will do, and it is the default action of IRAF's mscred ccdproc unless you specifiy otherwise). If these keywords are not present in your data, then you can add them in IRAF as follows.
 


The recomended parameters for the iraf ccdproc overscan subtraction (and trim) are
 

Overscan correction can also be done in PIPELINE, where you should be able to see the same parameters set in the /opt/cicada/config/iraf_table file.
 

Bias (or zero) subtraction : Create a zero correction using at least 7 bias frames from your run, after overscan subtraction and binning.


Apply the zero correction to all images


 

Details - Linearity

If you want your data reduced properly, you really should apply linearity corrections. (See the WFI Performance page for information on how lon-linear your data are). If you intend to apply a linearity correction you must do it after overscan/zero subtraction. Suggested polynomial corrections are available.

Unfortunately, IRAF does not offer linearity correction in its usual CCD processing. You can insert the derived linearity corrections into the file headers of each image extension as follows, and then apply them, but the process is quite inefficient.
 

hedit *.fits[1] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[1] LIN_A0 1.0      add=yes update=yes
hedit *.fits[1] LIN_A1 0.0      add=yes update=yes
hedit *.fits[1] LIN_A2 3.86e-12 add=yes update=yes
hedit *.fits[1] LIN_A3 0.0      add=yes update=yes

hedit *.fits[2] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[2] LIN_A0 1.0      add=yes update=yes
hedit *.fits[2] LIN_A1 0.0      add=yes update=yes
hedit *.fits[2] LIN_A2 3.18e-12 add=yes update=yes
hedit *.fits[2] LIN_A3 0.0      add=yes update=yes

hedit *.fits[3] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[3] LIN_A0 1.0      add=yes update=yes
hedit *.fits[3] LIN_A1 0.0      add=yes update=yes
hedit *.fits[3] LIN_A2 8.43e-12 add=yes update=yes
hedit *.fits[3] LIN_A3 0.0      add=yes update=yes

hedit *.fits[4] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[4] LIN_A0 1.02753     add=yes update=yes
hedit *.fits[4] LIN_A1 -1.6555e-6  add=yes update=yes
hedit *.fits[4] LIN_A2 1.7791e-11  add=yes update=yes
hedit *.fits[4] LIN_A3 0.0         add=yes update=yes

hedit *.fits[5] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[5] LIN_A0 1.0      add=yes update=yes
hedit *.fits[5] LIN_A1 0.0      add=yes update=yes
hedit *.fits[5] LIN_A2 4.26e-12 add=yes update=yes
hedit *.fits[5] LIN_A3 0.0      add=yes update=yes

hedit *.fits[6] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[6] LIN_A0 1.0      add=yes update=yes
hedit *.fits[6] LIN_A1 0.0      add=yes update=yes
hedit *.fits[6] LIN_A2 5.93e-12 add=yes update=yes
hedit *.fits[6] LIN_A3 0.0      add=yes update=yes

hedit *.fits[7] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[7] LIN_A0 1.0      add=yes update=yes
hedit *.fits[7] LIN_A1 0.0      add=yes update=yes
hedit *.fits[7] LIN_A2 -5.50957e-12 add=yes update=yes
hedit *.fits[7] LIN_A3 2.39334e-16  add=yes update=yes

hedit *.fits[8] LINCOM "Nt = Nm*(LIN_A0  +  LIN_A1*Nm + LIN_A1*Nm**2 +LIN_A3*Nm**3)" add=yes update=yes
hedit *.fits[8] LIN_A0 1.0      add=yes update=yes
hedit *.fits[8] LIN_A1 0.0      add=yes update=yes
hedit *.fits[8] LIN_A2 5.37e-12 add=yes update=yes
hedit *.fits[8] LIN_A3 0.0      add=yes update=yes


IRAF does not have a way to include linearity correction in its ccdproc processing. Nor is there a really useful linearisation routine. You have to use imexpr which is not bright enough to accept a list of input/output images. Furthermore, you can't make msccmd and imexpr interact in the way they should, so you have to split each MEF image into individual CCDs, then correct each individually, then join the images back together.

If an IRAF guru can provide a script/executable to speed this process up that would be incredibly useful. All advice gratefully received.

In principle, it should be straight-forward to put this a useful IRAF script. However, the IRAF scripting language is such an abortion I couldn't be bothered, so instead here is a recipe to make a giant list of commands which you can pipe into CL.

#
# Some csh to make a long file to linearise data in the directory
# /data1/cgt/FEB01/20010202/A/linearity/zeroed
# and put the results into the directory directly above.
#
echo mscred > doit
echo cd /data1/cgt/FEB01/20010202/A/linearity/zeroed >> doit
foreach j ( 02feb*.fits )
   echo mscsplit input=$j mefext=.$j:e delete=no verbose=yes >> doit
   foreach i ( 1 2 3 4 5 6 7 8 )
      echo "imexpr 'a*(a.lin_a0 + a.lin_a1*a + a.lin_a2*a**2 + a.lin_a3*a**3)' \
      output=l${j:r}_$i a=${j:r}_$i" intype=double outtype=real refim=a >> doit
   end
   echo imrename ${j:r}_0 l${j:r}_0 >> doit
   echo mscjoin input=l${j:r} output=../${j:r} delete=yes verbose=yes >> doit
   echo imdel ${j:r}_\*.fits go_ahead=yes verify=no default=yes >> doit
end
echo logout >> doit

cl < /data1/cgt/FEB01/20010202/A/linearity/zeroed/doit

Note that the imexpr processing is incredibly slow. However it does work. A dedicated routine which operated directly on MEF files would be much faster. It could also be optimised to only calculate terms for which the coefficient is non-zero.
 
 

Details - Dark subtraction

 

Details - Flat Fielding

 

IRAF MSCRED Processing Hints

IRAF MSCRED operations can be a bit obscure. I can't claim to be an expert, but here are a few things it took me ages to work out from the IRAF documentation.
Changing FITS keywords in your files. The trick is to know where the keywords are. Some keywords apply to the entire file (eg. FILTER). In MSCRED parlance they sit in the [0] extension. Other keywords apply to specific images (eg. TRIMSEC) and they sit in the [1],[2], ... extensions.

The tricky bit is if you try the command

hedit 02feb0166[1] FILTER "z #98 (WFI SDSS)"  upd+
in an attempt to update the filter name, MSCRED will TELL you it has changed it. But it won't have. You need to use
hedit 02feb0166[0] FILTER "z #98 (WFI SDSS)"  upd+
This one did frustrate me for days. I could NOT work out why I couldn't update a header entry. Note that
ccdhedit 02feb0166 FILTER "z #98 (WFI SDSS)"
will also not work.

How to linearise your data? See above.

How to put correct BIASSEC and TRIMSEC entries in your file header? See above.

Can't get your entire image to display in a 1 CCD pixel to 1 display pixel format? Edit your login.cl file and change the 'set stdimage' line to

       set stdimage = imt8800

but be aware your images will take somewhat longer to load!




This page created by Chris Tinney