Quick Summary | Software | Data Files | Reducing Data | Plotting Data | Automatic Reduction | Nod and Shuffle

Reducing SPIRAL Data using 2dfdr

Jeremy Bailey  - Jan 5 2001

1. Introduction

SPIRAL is an Integral Field Unit (IFU) spectrograph for the AAT. It consists of a lenslet array feeding light to a 32 by 16 array of fibres, which are then fed to a spectrograph giving 512 spectra which are imaged onto a 2048 by 4096 MITLL CCD. The IFU provides spectra over a 22 by 11 arcsecond rectangle of sky with 0.7 arcsec pixels. More information about Spiral can be found on the web at http://www.aao.gov.au/astro/spiral.html.

The data reduction system for the AAO 2dF multi-object fibre system (2dfdr) has been adapted to provide facilities for reducing Spiral data.

The software provides for extraction of spectra from the images, wavelength calibration, flat fielding, throughput calibration and sky subtraction. It also provides an interactive data display which can plot data as an IFU image at any wavelength, spectrum for any IFU pixel, or an image of spectra for each fibre number.

The ability to rapidly reconstruct an image from IFU data makes the system very useful for quick-look use at the telescope during observing. The reconstructed images can be helpful for target acquisition and centering.

2. Quick Summary of Spiral Data Reduction

1. Make a directory containing the data files for one spectrograph configuration.  (See Section 4) You might also need to set the classes of the data files.  (See Section 4.1)

2. Copy spiral.dat from DRCONTROL_DIR and edit to include your grating and wavelength setting.  (See Section 5.1)

3. Reduce a flat field to make a tram-line map. (See Section 5.2)

4. Plot the tram-line map and check it correctly matches the data. (See Section 5.3) If so go to step 6.

5. If the tram-line map doesn't match the data edit the spiral.dat file, exit from 2dfdr, restart it and go back to step 3. You may have to repeat several times to get it right. (See Section 5.4)

6. Turn off the "Rotate and Shift to Match" option in the Extract parameters page (See Section 5.5)

7. Reduce an arc. (see Section 5.6)

8. Plot the reduced arc and check that the lines have been correctly identified. (see Section 5.7)

9. Reduce any further arcs. (See Section 5.6)

10. Reduce offset or twilight skies if you wish to use them for fibre throughput calibration. (See Section 5.8)

11. Reduce your object files. (see Section 5.9)

12. Use the plot option to examine your data. (see section 6)

13. If you have several reduced files for the same object, combine them. (see section 5.10)

Steps 7 and 11 can be done in automatic mode (see Section 7).

3. The Software

3.1 Where to Find It

2dfdr is available for Sun Solaris and Linux. It is available via the web at http://www.aao.gov.au/2df/software.html. From version 2.0 the standard 2dfdr contains all the features needed for reducing Spiral as well as 2dF data.

2dfdr is installed on the AAO solaris machines at Epping and at the telescope. At Epping it is found in /prog/jab/2dfdr and at the telescope it is in /epping/jab/2dfdr.

3.2 Starting Up

The procedure for running the software is as follows:

  > setenv DRCONTROL_DIR /prog/jab/2dfdr

This is the command to use at Epping. At the telescope use /epping/jab/2dfdr, at other sites use whatever directory you have put the 2dfdr software in.

  > source $DRCONTROL_DIR/2dfdr_setup

This sets up everything needed for running 2dfdr.

  >  drcontrol spiral.idx &

This starts up 2dfdr in its Spiral mode. The 2dfdr main window should come up and in its title bar it should say "Spiral Data Reduction" followed by a version number and date. Two windows marked "DRPLOT1 - Diagnostic Plots" and "DRPLOT2 - General Plots" should also appear.

2dfdr is closed down by selecting EXIT from its file menu.

Another important command is "cleanup". You should type cleanup if you have any problems starting or closing down 2dfdr. Cleanup releases any resources which might have been allocated by the Drama software on which 2dfdr is based.

2dfdr can fail to start properly if there are insufficient colours available on an X windows display. You should therefore close down any applications that use many colours (such as netscape) before running 2dfdr.

3.3 Restarting 2dfdr

If you are working on a set of data, and have to shut down 2dfdr and return to it later, start up 2dfdr as normal. Then, click on the Setup button in the Automatic Reduction section in the top right of the window. This causes 2dfdr to check all the files in the directory and take note of all the calibration files which were reduced in the previous session. It effectively restores 2dfdr to the state it was in when the session was interrupted.

If you don't do this 2dfdr won't know about previously reduced calibration files (tram line maps, arcs etc.) and so may not properly reduce subsequent runs.

3.4 Starting from Scratch

If you get into trouble and want to go back to the beginning and commence your reduction from scratch do the following.

1.  Exit from 2dfdr

2.  Delete all reduced files and tram line maps by using the following commands:

>   rm *red.sdf
>   rm *tlm.sdf

3.  Restart 2dfdr by typing:

>   drcontrol spiral.idx &

4. Data Files

2dfdr makes use of raw data files from the AAO Observer system which are in Starlink's NDF format and have a file extension of ".sdf".

If your data has been provided in FITS format (e.g. from the AAT archive) you will need to convert it to NDF for use. This must be done using the fits2ndf command of the Starlink Convert package. This program is provided with the 2dfdr release. To convert a file use the command:

>   fits2ndf run0001.fts run0001 \\

For Spiral data redution you will need the following data files.

You should make a directory containing a set of related files all taken with the same spectrograph configuration (i.e. grating and central wavelength). If you used different spectrograph configurations on the same night then these files should be split up into a separate directory for each configuration.

2dfdr uses the INSTRUME keyword in the FITS header to recognize the data as Spiral data. It should have the value "Spiral" in the first six characters. It will be set correctly if the INST command in the observer system was used. If this was not done you will have to add this item to the header for the data reduction to work correctly. This can be done using the Figaro fitset command.

4.1 Data File Classes

2dfdr needs to recognise which data files are flat fields, arcs etc. in order to process them appropriately. It will do this automatically if you used the correct command on the Observer system when you took the data. (i.e. FLAT to take flat fields, ARC to take arcs, SKY to take offset or twilight skys, and RUN for object data).

If this wasn't done 2dfdr can be used to assign a class to each file. This is done by picking the option Set Class... from the Commands menu, selecting the file, and then using the menu to specify a class for it. The classes to use are as follows:
MFOBJECT object files
MFFFF  flat fields
MFARC arc files
MFSKY twilight or offset sky data

5. Reducing Data

The data reduction system is designed to be essentially automatic once things have been set up. It can operate in an automatic mode in which case it will go through all the files in a directory in an appropriate sequence, or can be used in a manual mode in which case each file is reduced individually.

With Spiral data it is necessary to reduce at least the initial flat field in manual mode. Remaining files can then be reduced in either manual or automatic mode.

5.1 The spiral.dat File

Once you have set up a directory containing your files the remaining thing you need is a file called spiral.dat which provides the software with information on the instrument configuration. You can copy a default version of spiral.dat from the DRCONTROL_DIR directory and then edit it.

>  cp $DRCONTROL_DIR/spiral.dat .

You need to edit the first few lines of this file to specify the spectrograph configuration.

GRATLPMM    1200
ORDER       1
LAMBDAC     6630
     1        2       P
     1        3       P
     1        4       P
     1        5       P
     1        6       P
     1        7       P
     1        8       P

GRATLPMM needs to be set to the grating lines per mm for the grating in use and LAMBDAC should be the central wavelength of the region being observed in angstroms. LAMPNAME is the name of the calibration lamp used (usually Neon). The software uses this to determine which arc line list to use. Possible lamp names supported by the software and the corresponding arc line files are:
CuAr cuar.arc
Helium helium.arc
Helium+CuAr hecuar.arc
FeAr fear.arc
Helium+FeAr hefear.arc
LDSS ldss.arc
Neon neon.arc

The names must be exactly as written above to be recognized by the software.

The remainder of the spiral.dat file lists the IFU fibres giving the pixel coordinates on the IFU for each fibre along the slit.

5.2 Reducing a Flat Field to Make a Tram-Line Map

The next stage is to reduce the fibre flat field to be used for tracing the fibre spectra across the CCD. This makes a file called a tram-line map, which is used in all subsequent reductions. Choose REDUCE from the Commands menu and select the name of the flat field file. It will start to reduce it on one of the two reduction tasks (DREXEC1 or DREXEC2). Click on the tab of which ever of these two tasks goes to Busy to follow the progress of the reduction.

The tram-line map is created by taking a cut up the centre of the image and finding the peaks in the data which correspond to the individual fibres. It traces a curved line through each of these peaks across the CCD, and then makes a rotation adjustment to match this to the actual data. The curvature is due to the distortion in the spectrograph optics which is fitted by a radial distortion model specified by parameters in the Extract parameters page. The default values should normally be OK.

5.3 Is the Tram-Line Map OK?

If the spectrograph is in good focus, the number of peaks found in this process should be close to 509. You will see the number actually found reported on the terminal window from which you started 2dfdr in a line saying something like "509 peaks found". Although there should be 512 fibres in the IFU, there are actually 3 which are never useable, one because it is broken and two because they are too close to their neighbours. Often less than 509 will be found since there are a few fibres which are sometimes missed as they are close to others. If, however, the number of peaks found falls to less than 500, this normally means the focus is poor and the spectrograph should be refocused. A problem that is encountered particularly with high resolution gratings is that the best spectral focus is different to the best spatial focus. You will have to choose a compromise focus which allows the data reduction system to locate the fibres while still giving acceptable arc line widths.

The number of peaks found in the data must match the number of fibres listed in the spiral.dat file (509 in the default version). The spiral.dat file should also correctly list the IFU coordinates of each peak found in the data. If it doesn't you will have to edit the spiral.dat file to remove the fibres which are being missed in the peak finding process. To find out which fibres have been missed you need to plot the tram-line map overlaid on the data. Do this by selecting Plot Tram Map... from the Commands menu. Select the name of the tram line map file which will be the original data file with a tlm appended. For example, if the flat field file was run0003.sdf the tram line map will be in run0003tlm.sdf.

To see the tram lines clearly the display needs to be zoomed several times by using the Z key. Other keys which can be used adjust this display are O to zoom out, and H and L to set the high and low scaling levels. The Next and Prev buttons step up and down through the zoomed image. By examining this plot it should be possible to work out which fibres have been missed. The plotted tram-lines are labeled with their X, Y coordinates on the IFU. There are usually noticeable gaps between each slit block of 32 fibres and the next and these should line up correctly with the numbering. The procedure is best illustrated by an example:
In this example the fibre number 32,2 (that is number 32 in slit block 2) has been missed by the peak finding algorithm since it is too close to number 31,2. The two fibres overlay each other and appear as a brighter image before the dark gap between slit block 2 and slit block 3. Fibre 1 in the slit block 3 has been incorrectly labeled as 32,2. To correct this fibre 32,2 must be deleted from the spiral.dat file. Note that in spiral.dat the X and Y coordinates are in the opposite order so the line to be deleted will read  "2      32".

As well as the slit block gaps there are a number of other distinctive features which can help to match the fibre numbering with the image. For example fibre 8,2 is missing, there is a group of three fibres close together at the start of slit block 13. The following image of the slit blocks helps to show these features (click on it for an expanded view):

5.4 Editing the Spiral.dat file

In this way the spiral.dat file must be modified until it lists only those fibres which the peak finding algorithm finds in the data.

To test out a modified spiral.dat file it is necessary to exit from 2dfdr, restart it and reduce the flat field again (if you don't exit from 2dfdr it won't make a new tram-line map but will reuse the old one - to be sure this doesn't happen it is a good idea to delete the old tram line map file as well).

The end result should be a spiral.dat which gives you tram-lines exactly matching the data. The best test of this is that the slit block gaps visible as dark gaps in the data match the numbering of the slit blocks on the overlaid tram lines all the way across the CCD. It may take several attempts to get this right.

5.5 Continuing with the Reduction

Once a good tram-line map has been produced it is possible to continue with reducing other types of files. There is one important thing to do before moving on.

Enter the Extract parameters page and turn OFF the option "Rotate/Shift to Match".

If this is not done the software will attempt to adjust the tram line map in shift and rotation to match each data file. This often doesn't work well with Spiral data as there may only be signal in a few fibres - for example when observing a bright star - and so there isn't enough data for the matching to work on. It is in any case unnecessary since Spiral is a bench mounted spectrograph, so the tram-line pattern shouldn't move on the CCD.

At this stage it is now possible to put 2dfdr into its automatic mode and let it go through and reduce all remaining files. However it is probably better to stay in manual mode at least to reduce the first arc, so that you can check it is identifying lines correctly.

5.6 Reducing Arcs

The next stage is to reduce the arc (wavelength calibration lamp) files. To reduce a file the procedure is exactly the same as for the flat field. Use the Reduce... option in the Commands menu and select the name of the file. When the arc has finished reducing use the plot button to plot the reduced arc. It should look something like this:

The arc lines should be straight lines running up and down the image, and at the correct wavelengths.

5.7 Problems with Arc Fitting

If the arc line plot doesn't show the nice straight lines seen in the above picture there are a couple of things which could be going wrong.

1. The central wavelength or grating information in the spiral.dat file could be wrong.

2. The arc line list being used may not be a good representation of the lines present in the data.

You should get the central wavelength in the spiral.dat file as close as possible to the true central wavelength of your data. Not only does this ensure that the automatic arc identification and fitting work well, but it is the centre actually used for the final rebinned data, so if it is wrong you will lose a bit of data at one end of the spectrum.

If you decide to change the wavelength in the spiral.dat file, you must start your reduction again from the beginning - exit from 2dfdr, delete any tram-line map (*tlm.sdf) files, restart 2dfdr and reduce the flat field and then the arc. This is because wavelength information is included in the tram-line map file.

If you still get problems with the correct central wavelength the arc line list may be the problem. The line list files are stored in DRCONTROL_DIR, but you can override the standard ones by having a copy in your own working directory. This can be useful.

For example if you used a non-standard arc you can make a line list to match this. We have, for example, taken data with a combination of a Neon and HgCd lamp. Too reduce this we made a local copy of the neon line list (neon.arc) and edited it to add in the Hg lines in our region of interest. The line lists contain the wavelength of each line followed by a rough intensity (this is just used by the software to select the strongest lines). The only restriction is that the file must have one of the standard names described in section 5.1.

When an arc is reduced a text file describing the arc fits is produced. For example if the arc is run0006.sdf, the text file will be arclist006.dat. Examining this file may indicate what is going wrong. For example if a particular line is consistently being misidentified, deleting this line from the line list may fix the problem.

5.8 Offset or Twilight Sky Data

Twilight sky or offset sky data (class MFSKY) can be used to calibrate the relative fibre throughput of the IFU. These data are reduced in the same way as other data files. If there were a number of sky files they will be automatically combined and the throughput derived from the combination. Once a throughput map has been made in this way it will be used to throughput calibrate any subsequent object frames. You can turn off this throughput calibration in the SkySub parameters page.

There is another method of throughput calibration  using sky lines in the object frames. If you use this method you don't need sky exposures.

5.9 Reducing Object Data

Once all calibration frames have been reduced, the object data can be reduced. When each data frame is reduced the software will automatically choose the closest matching (in time) calibration data (arcs, throughput maps etc.) to use with it. If you want to check exactly which calibration was used for a particular file this is available using the Show History button.

There are a number of options you might want to select for reduction of object files:

1. Flat fielding using a fibre flat - This is turned on from the General parameters page (note all the other options in that page which are on by default normally have no effect since we don't normally take bias and dark frames, and have no way of taking a full chip flat).

2. Throughput calibration - This is specified in the SkySub parameters page. It is on by default, but if you don't have sky exposures for throughput calibration you can try using the sky line method by selecting SKYLINE(KGB) in the menu. This measures the fibre throughput based on the strength of sky lines in the object data itself.

3. Sky Subtraction - Again this is on by default, but will only have any effect if you specify some fibres as sky fibres in the spiral.dat file. To do this replace the P with an S on the lines for each IFU pixel you want to use as sky. A combined sky spectrum will be derived from all the sky fibres and subtracted from every spectrum on the array. For sky subtraction to work well the data must be throughput calibrated.

5.10  Combining Reduced Data

If you have taken several exposures of the same object you can combine these by using the Combine Reduced Runs... option in the Commands menu. This combination is done in such a way that cosmic rays are rejected. There are some parameters controlling this combination in the Combine parameters page. If you have high S/N data you may find you need to increase the Rejection Threshold parameter. Otherwise good data such as strong emission lines, may be rejected as cosmic rays.

6  Plotting Data

6.1  Plot Modes

A reduced data file can be plotted using the plot button in the Data section of the user interface, or the Plot... option in the Commands menu. The display of reduced data has three modes. By default it comes up in fibre mode as shown below:

This image shows the fibre spectra for each of the IFU fibres. It is possible to interact with the plot using the cursor and keyboard as well as the buttons on the left of the plot window. For example typing the Z key zooms in by a factor of 2 around the point indicated by the cursor. The H and L keys set the high and low scaling levels to the value of the point indicated by the cursor, which can be a pixel in the image, or in the scale bar on the right.

The X key plots a spectrum for a single fibre at the position indicated by the cursor, changing the mode to the spectrum mode as shown below:

This plot shows a single fibre spectrum. The 15 7 in the label are the coordinates of the fibre in the IFU.

The third mode is the IFU image mode which is obtained by using the I key with the cursor indicating the required wavelength for the image on a spectrum or fibres mode plot. It looks like this:

It is a 32 by 16 pixel image of the data from the IFU for a single wavelength. From the image it is possible to plot a spectrum for any image pixel by using the "." key.

Here is a full list of the keys used for interacting with the plot:
C Plot image as contour plot
G Plot image as greyscale plot
F Plot image as false colour plot
H Make value at cursor position the high scaling level
L Make value at cursor position the low scaling level
Z Zoom in by factor of 2 on cursor position
O Zoom out by factor of 2 on cursor position
P Centre plot on cursor position
X Plot cut through cursor position in X
Y Plot cut through cursor position in Y
I Plot IFU image at cursor wavelength
. Plot spectrum of pixel in IFU image
R Restore original plot size and scaling
[ ] Select corners of region to expand
A Report centroid and data value within aperture
S Show current aperture size
< > Increase or decrease aperture size
Q Quit
Any other Report cursor position and data value
6.2  IFU Image for a Range of Wavelengths

It is also possible to plot an IFU image for a range of wavelengths, but this can't be done from within the plotting program. Go to the Plots parameters page and set the start and end wavelength. Then select "IFU" from the Plot Type menu. Then plot the data using the Plot... option in the menu or the Plot button in the Data section.

6.3 Saving an IFU Image as an NDF file

Every time you quit from a plot of an IFU image the image will be saved in an NDF file called by
default ifu_image.sdf - the name can be changed using the Plots paprameter page.

6.4 Hardcopy Plots

There are two ways of doing this. First you can use the Print... option in the File menu of the plot windows to print whatever is displayed on the screen or save it as a postscript file.

The second way is the use the Hardcopy Plot... item in the Commands menu or the Plot button with the Hardcopy option selected. These two methods are equivalent. The Hard parameters page sets some of the options for hardcopy plots. There is one option here which is not available in the screen plotting and that is to plot a range of fibre spectra at up to 20 per page. The hardcopy plots produced in this way appear as postscript files of the form gks74.ps or gks74.ps.n (where n is a sequentially allocated number).

7. Automatic Reduction

You can use the automatic mode of 2dfdr to reduce all your files after the initial flat field. You must have first reduced a flat field to give a tram-line map, and then turned off "Rotate/Shift to Match" in the Extract parameters page.

Then go to the Automatic Reduction section in the top right of the window and click the Setup button. this will cause the system to search your directory for data files and take note of their types. The data is then accessible via the Data section in which you can select files by name or run number.

Next make sure that you have all the parameters correctly set for reducing your data.

Then click the Start button in the Automatic Reduction section. The software will now go through and reduce all your files in an appropriate sequence. That is it will reduce calibration files such as arcs first and files that need the calibration such as object data last.

8. Reducing Nod and Shuffle Data

2dfdr from version 2.1 can also automatically handle data taken with the nod and shuffle mode of Spiral. In this mode a mask is placed over the IFU so that only the central 15 fibres of each slit block are illuminated. This provides spaces between each slit block in to which charge can be shuflled during the sky exposure.

The only change from the normal reduction procedure is that a modified spiral.dat file is needed contaning only the 15 by 16 fibres illuminated by the mask rather than the 32 by 16 fibres of the full IFU. The spiral.dat file also needs to contain one extra parameter in it header section.

GRATLPMM    1200
ORDER       1
LAMBDAC     6630
SHIFT       63
     1        1       P
     1        2       P
     1        3       P
     1        4       P
     1        5       P
     1        6       P
     1        7       P
     1        8       P

This is the SHIFT parameter and specifies the charge shuffling shift in binned pixels. It is therefore half the shift value specified on the charge shuffling user interface. Flat fields and arcs should be taken as normal with the mask in place, but without charge shuffling. These can then be reduced as normal. An object exposure taken in charge shuffling mode will be recognized by the METHOD item in its FITS header and 2dfdr preprocesses it by subtracting from every pixel, the corresponding pixel SHIFT pixels away. In this way the sky subtraction is done before the data are extracted from the image. The remainder of the reduction proceeds as normally, but the resulting image will show essentially perfect sky subtraction.

Data taken with the mask but without nod and shuffle will also be correctly reduced by the system.