The 2dfdr data reduction package uses FITS format for input, output and internal manipulation of files. FITS (Flexible Image Transport System) is the nearly universally accepted file format for astronomical data endorsed by the IAU. An overview of FITS with links to reference documents is available at the NASA FITS webpages.
When analysing data from an observing run, one needs to map the combined spectra returned from the reduction task back to individual objects from the input catalogue. All of the relevant information is contained within the combined output file(s).
This page documents the format of FITS files used and written by the
2dfdr data reduction software. First the various file types are explained,
and then the internal file extensions are discussed.
- Summary of the 2dfdr Output File Format
- File Types
- File Parts
The table below gives a summary of the 2dfdr output file content, for either the individual frame *red.fits files or a
|Primary image||.fits||The primary extension in the .fits file is a WxN image containing the number of pixels in each spectrum and N is the number of spectra represented. This is 400 for AAOmega data (392 science fibres and 8 guide fibres). Unused science fibres and Sky spectra, are included in the output file along with the guide fibre spectra, even though the spectra contain no information. In the case where multiple sets of AAOmega datasets, which contained a subset of common objects, have been combined, the format is a little more complex.|
|First Extension||.fits||The variance extension is also a WxN array identical in size to the primary extension. Each member contains the variance for the corresponding element in the primary extension.|
|Second Extension||.fits||FITS binary table, with N rows, one for each fibre. Each Row contains information for each object such as RA, Dec, 2dF Pivot number and more|
|Other Extensions||...||The files contain several other extensions which are generally only used when deeper analysis of the data is required. They are not necessarily in order and are accessed by name.|
|For example:||Sky spectrum
Stored as a FITS binary table, with W rows.
Each row contains the one element of the 'typical' sky spectrum used in the data reduction ('typical' because for a combined frame it is not obvious how the final sky spectrum for each fibre should be represented here).
Note: The variance information is correctly propagated, the sky spectrum is not presented here for this purpose.
redsuffix indicates it is a reduced file. Other suffixes are
imfor image files,
exfor extracted files and
tlmfor tramline map files. Files without such a suffix are normally raw files. Filenames have four digits immediately preceding the type suffix. These four digits are the exposure sequence number, and usually begin with
0001each evening. The digit preceding the sequence number in the example indicates which the AAOmega spectrograph was used,
1for blue and
- had bad pixels marked,
- the overscan bias region has been processed, subtracted and removed,
- any cosmic rays have been removed (if requested),
- it has been divided by the long-slit flat frame (if requested), and
- the bias frame has been subtracted (if requested).
imto the sequence number of the corresponding raw file. This gives names like
exto the sequence number of the corresponding raw file. This gives names like
redto the sequence number of the corresponding raw file. This gives names like
31jan10083red.fits. SIGMAPRF HDU array of the same size containing the sigma of the Gaussian profile. Tramline map files have names that are formed by suffixing "tlm" to the sequence number of the corresponding raw file. This gives names like
- FITS file parts diagram:
rawfiles), processed CCD image (
imfiles), or spectral data (
redfiles). Images are the same size as the CCD. Raw images are slightly larger as they contain the overscan bias region. Spectral data is dimensioned wavelength (number of pixels in spectral direction) by the number of fibres. The data type is 16-bit integer for
rawfiles produced by the instruments, and 32-bit IEEE floating point for all data files produced by 2dfdr. Where Used: Every file has this HDU. It is written and read by 2dfdr.
The fibre binary table lists, for each fibre, the columns listed in the table below:
|1||NAME||String||Object name from the configure .fld file|
|2||RA||Real||Right Ascension from the configure .fld file|
|3||DEC||Real||Declination from the configure .fld file|
|4||X||Integer||2dF field plate X co-ordinate (in microns)|
|5||Y||Integer||2dF field plate Y co-ordinate (in microns)|
|6||XERR||Integer||Reported error in X in final fibre placement|
|7||YERR||Integer||Reported error in Y in final fibre placement|
|8||THETA||Real||Angle of fibre on field plate|
|9||TYPE||Character||Fibre type: F-guide, N-broken, dead or no fibre, P-program (science), S - Sky, U-unallocated or unused|
|10||PIVOT||Integer||2dF fibre pivot number|
|11||MAGNITUDE||Real||Object magnitude from the configure .fld file|
|12||PID||Integer||Program ID from the configure .fld file|
|13||COMMENT||String||Comment from the configure .fld file|
|14||RETRACTOR||Integer||2dF retractor number|
|15||WLEN||Real||Wavelength from the configure .fld file. This column was added around 2005|
|16||EXPOSURE||Integer||This column may appear in some combined output files where the combined datasets contained a subset of common objects and therefore exposure times differ for different objects. The column gives the exposure time in seconds for the fibre.|
There are two very important, and very different, numbers which one must understand in order to recover the information on which object each fibre was allocated: Fibre slit position AND 2dF Fibre-Pivot position. For the most part there is a one-to-one correspondence between these numbers. Usually the fibre at AAOmega slit position 1 (bottom of the CCD image) will map directly to 2dF Pivot position 1, and 400 will map to 400 (note, 400 is a guide fibre and so maps to a blank space at the top of the CCD image). However, during manufacture or repair of each of the AAOmega slit units, it is sometimes possible for the order of fibres in each of the AAOmega slits to fall out of synchronization with the 2dF Pivot numbering. It is not practical to mechanically alter either position so each of the two fibre numbers (slit position and Pivot position) are propagated in the fibre table.
In the primary image (and also the variance array, stored in the first extension) the fibre at the bottom of the image, which is the fibre at slit position 1, corresponds to the first row in the fibre table (the second fits extension). The table contains a column entry, PIVOT, which gives the 2dF pivot position for this fibre. This is the fibre number seen by the configure software. The very top fibre in a CCD image corresponds to the very last entry in the fibre table (which will be an AAOmega guide fibre in the case of a single AAOmega data set). There is ALWAYS a one-to-one correspondence between each spectrum position in the image and the fibre table. There is typically a one-to-one correspondence between slit position and 2dF pivot position but with a number of known mismatches and discontinuities which are tracked via the PIVOT column of the fibre table.
This list is not exhaustive.
One can save a list file (file menu -> ..list) which contains the allocated 2dF Fibre-Pivot number for each allocated fibre. Note, this is the Pivot number for 2dF NOT the fibre number in the reduced 2D spectra file.
The 2dfinfo procedure comes packaged with 2dfdr. It can be used to recover information on the fibre from the .fits file. The syntax for the command is:
2dfinfo file.fits <option>
If the <option> is omitted then the list of options is given. To recover the fibre table information one would use:
2dfinfo file.fits fibres
The IRAF/STSDAS package TABLES has a number of routines designed for manipulating tables. A simple example might be:
IRAF> tdump combined_frame.fits > output.txt
This would create a complete listing of the fibre binary table information and pipe it to an ascii text file. Formatting the output can be achieved with:
IRAF> tprint combined_frame.fits columns="NAME,RA,DEC" > output.txt
For users of IDL, the NASA IDL astronomy library has some excellent FITS data access routines.
Starting from a combined FITS frame, combined_frame.fits, one might use the following code extracts to manipulate AAOmega data. Note, there are cleverer (and quicker) ways to perform the operations below with the NASA astrolib tasks, the code here is given as a simple example.
;; Read in the spectral image, store the header information
;; And the variance array
;; Make a wavelength vector, note the use of CRPIX1, which is often not expected by many users.
;; If missed, the wavelength solution will tend to be wrong by half a CCD width
crpix=fxpar(header0,'crpix1')-1.0 ; The -1.0 is needed as IDL is ZERO indexed
;; Read in the object identification information
;; And read a copy of the sky spectrum subtracted from the data.
;; Note, for a combined frame, this is the sky spectrum from the first file in the list of combined frames.
;; It is a good representative sky spectrum, but should be used with caution for the combined spectral data.
fxbclose,unit fv, ds9, etc.) to describe the axis so 2dfdr output files are correctly handled by these viewers. Within the 2dfdr code, axis information is generally held in a one-dimension vector of 32-bit IEEE floating point values. The vector values are constrained to always being linear since only 3 keywords are used. The transformation from FITS keyword values to vector, and vice versa, is hidden in the
TDFIO_AXIS_WRITEroutines, respectively. History: The internal vector format comes from when 2dfdr used Starlink's Extensible N-Dimensional Data Format (NDF). The NDF "CENTRE" axis component was used. This was a vector holding the pixel centre coordinates, with a separate vector for each axis. The NDF axis attributes of LABEL and UNITS were passed to the FITS keyword string values CTYPEn and CUNITn, respectively. N.B. The functionality of this HDU is nearly identical to the WAVELA Extension, and many of the same issues apply. One of the two should/could (probably) be eliminated. Where Used: Every file has this information. Within 2dfdr, axis information is used when plotting files (read only). The abscissa values (wavelength) are set during scrunching and splicing. They are also used to judge wavelengths of interest when matching arc peaks and known wavelength intensities (read only). Axis Information, and many of the same issues apply. One of the two should/could be (probably) eliminated.
- FIBNO is the fibre number, where 1 <= FIBNO <= instrument fibre count
- COEFF is the coefficient number, where 1 <= COEFF <= MAX_SHIFTS (defined in tdfio.inc as 10). That is, there is a set of 10 coefficients for each fibre.
- During the spectra processing for red(uced) arc frames (see
reduce_arc.f). These values are used to rebin fibre flat and science frames.
- During spectra processing for red(uced) sky frames (see
reduce_sky.f).These values are used to rebin science frames.
||Fibre flat field image, raw and im(age)|
||Fibre flat field spectra, ex(tracted) and red(uced)|
||Arc image, raw and im(age)|
||Arc spectra, ex(tracted) and red(uced)|
||Science image, raw and im(age)|
||Science spectra, ex(tracted), red(uced), combined and spliced|
||Flux calibration image, raw and im(age)|
||Flux calibration spectra, ex(tracted) and red(uced)|
||Bias frame image, raw and red(uced)|
||Long-slit flat image, raw and red(uced)|
Sarah Brough (sb_at_aao.gov.au)
Ron Heald (rwh_at_aao.gov.au)