The 2dfdr data reduction package uses FITS format for input, output and internal manipulation of files. FITS (Flexible Image Transport System) is the nearly universally accepted file format for astronomical data endorsed by the IAU. An overview of FITS with links to reference documents is available at the NASA FITS webpages.
When analysing data from an observing run, one needs to map the combined spectra returned from the reduction task back to individual objects from the input catalogue. All of the relevant information is contained within the combined output file(s).
This page documents the format of FITS files used and written by the
2dfdr data reduction software. First the various file types are explained,
and then the internal file extensions are discussed.
Contents:
Summary of the 2dfdr Output File Format
The table below gives a summary of the 2dfdr output file content, for either the individual frame *red.fits files or a
| IRAF Format | Contents | |
| Primary image | .fits[0] | The primary extension in the .fits file is a WxN image containing the number of pixels in each spectrum and N is the number of spectra represented. This is 400 for AAOmega data (392 science fibres and 8 guide fibres). Unused science fibres and Sky spectra, are included in the output file along with the guide fibre spectra, even though the spectra contain no information. In the case where multiple sets of AAOmega datasets, which contained a subset of common objects, have been combined, the format is a little more complex. |
| First Extension | .fits[1] | The variance extension is also a WxN array identical in size to the primary extension. Each member contains the variance for the corresponding element in the primary extension. |
| Second Extension | .fits[2] | FITS binary table, with N rows, one for each fibre. Each Row contains information for each object such as RA, Dec, 2dF Pivot number and more |
| Other Extensions | ... | The files contain several other extensions which are generally only used when deeper analysis of the data is required. They are not necessarily in order and are accessed by name. |
| For example: | Sky spectrum Stored as a FITS binary table, with W rows. Each row contains the one element of the 'typical' sky spectrum used in the data reduction ('typical' because for a combined frame it is not obvious how the final sky spectrum for each fibre should be represented here). Note: The variance information is correctly propagated, the sky spectrum is not presented here for this purpose. |
File Types
Introduction
The 2dfdr software creates, reads and writes several file types. The possible types are listed here. A file type can be distinguished by its filename and the type suffix. An example is the AAOmega file31jan10083red.fits. The red suffix indicates
it is a reduced file. Other suffixes are im for image
files, ex for extracted files and tlm for
tramline map files. Files without such a suffix are normally raw
files.
Filenames have four digits immediately preceding the type
suffix. These four digits are the exposure sequence number, and
usually begin with 0001 each evening.
The digit preceding the sequence number in the example indicates which
the AAOmega spectrograph was used, 1 for blue and
2 for red.
raw
These files are produced by the instrument. They are always read-only to 2dfdr, i.e. they are never modified by the reduction software.im(age)
This is the raw file that has- had bad pixels marked,
- the overscan bias region has been processed, subtracted and removed,
- any cosmic rays have been removed (if requested),
- it has been divided by the long-slit flat frame (if requested), and
- the bias frame has been subtracted (if requested).
im to
the sequence number of the corresponding raw file. This gives names
like 31jan10083im.fits.
ex(tracted)
An ex(tracted) file has had intensity information extracted from the image file. This is done for each fibre used in the exposure. The spectrum of each fibre is given producing a Wavelength by fibre Number (WxN) array. Extracted files have names that are formed by suffixingex to the sequence number of the corresponding raw
file. This gives names like 31jan10083ex.fits.
red(uced)
This is the final reduced file. It contains a Wavelength by fibre Number (WxN) array. It is produced by applying observation type (fibre flat, arc, science, etc.) specific algorithms to the ex(tracted) file. Reduced files have names that are formed by suffixingred
to the sequence number of the corresponding raw file. This gives names
like 31jan10083red.fits.
tlm
This is the tramline map file. It is normally produced in the same step as the ex(tracted) file. It provides the centre fibre positions on the image in pixel units in its PRIMARY array. It may contain a SIGMAPRF HDU array of the same size containing the sigma of the Gaussian profile. Tramline map files have names that are formed by suffixing "tlm" to the sequence number of the corresponding raw file. This gives names like31jan10083tlm.fits.
combined reduced
This is the result of combining two or more red(uced) files. 2dfdr uses the namecombined_frames.fits.
spliced reduced
This is the result of splicing a pair of AAOmega red and blue reduced files. 2dfdr uses the namespliced.fits.
combined BIAS
Sometimes called the "master" bias file, this is the result of combining two or more reduced bias files. 2dfdr uses the nameBIAScombined.fits.
combined LFLAT
This is the result of combining two or more reduced long-slit flat files. 2dfdr uses the nameLFLATcombined.fits.
diagnostic
2dfdr produces various files used to diagnose particular operations on the data.File Parts
Introduction
Each FITS format file contains one or more Header Data Units (HDU). HDUs outside the primary HDU are known as "extensions". There are three types of HDU: image, ASCII table or binary table. 2dfdr uses only image HDUs with a binary table HDU used to hold the fibre information. The FITS standard requires each HDU to have a header (keyword) section and data section. Each header section contains any number of keyword name, value and comment 80-character records. The purpose of the keyword section is to provide information about the connected HDU data section, and the FITS format specifies a minimum set of keywords that each HDU must have (SIMPLE, BITPIX, NAXIS, NAXISn where n is 1 to array rank, and END).- FITS file parts diagram:
[0] Primary
A 2-dimension image array holding the raw CCD image (raw files), processed CCD image (im files),
or spectral data (ex and
red files). Images are the same size as the CCD. Raw
images are slightly larger as they contain the overscan bias
region. Spectral data is dimensioned wavelength (number of pixels in
spectral direction) by the number of fibres. The data type is 16-bit
integer for raw files produced by the instruments, and
32-bit IEEE floating point for all data files produced by 2dfdr.
Where Used: Every file has this HDU. It is written and read
by 2dfdr.
[1] VARIANCE
A 2-dimension image array holding the expected image data variance. The array is identical in size to the primary array, and each member contains the variance for the corresponding primary array member. The data type is always 32-bit IEEE floating point. Values are initially derived from the image data along with values determined from photon statistics, and the detector read noise and gain. The variance is then propagated and adjusted through subsequent steps of the processing. Where Used: All 2dfdr output files have this HDU. Raw files do not have this HDU. It is written and read by 2dfdr.[2] FIBRE TABLE
A 2-dimension binary table containing a row for each fibre. Each row describes how the corresponding fibre was used in the observation. The table columns are character data (e.g. the name of the astronomical object the fibre was observing), integer data (e.g. the fibre number) and floating point data (e.g. the object right ascension and declination). Descriptions for all columns are below. The table appears in raw instrument-produced files where it is created during the observation when it is filled with information output by the "configure" program. The table is then copied from the raw file to all 2dfdr output files. There are two types of fibre tables that are identified by their names, either "FIBRES" or "FIBRES_IFU". FIBRE_IFU tables are produced only by the AAOmega IFU instrument and they have slightly different columns to other instruments. Where Used: Every file has this HDU. It is READ ONLY to 2dfdr with one exception. The exception can occur when reduced files using different configurations are combined. In this case rows are added to the fibre table.Fibre Table Columns
The fibre binary table lists, for each fibre, the columns listed in the table below:
| Column | Name | Type | Description |
| 1 | NAME | String | Object name from the configure .fld file |
| 2 | RA | Real | Right Ascension from the configure .fld file |
| 3 | DEC | Real | Declination from the configure .fld file |
| 4 | X | Integer | 2dF field plate X co-ordinate (in microns) |
| 5 | Y | Integer | 2dF field plate Y co-ordinate (in microns) |
| 6 | XERR | Integer | Reported error in X in final fibre placement |
| 7 | YERR | Integer | Reported error in Y in final fibre placement |
| 8 | THETA | Real | Angle of fibre on field plate |
| 9 | TYPE | Character | Fibre type: F-guide, N-broken, dead or no fibre, P-program (science), S - Sky, U-unallocated or unused |
| 10 | PIVOT | Integer | 2dF fibre pivot number |
| 11 | MAGNITUDE | Real | Object magnitude from the configure .fld file |
| 12 | PID | Integer | Program ID from the configure .fld file |
| 13 | COMMENT | String | Comment from the configure .fld file |
| 14 | RETRACTOR | Integer | 2dF retractor number |
| 15 | WLEN | Real | Wavelength from the configure .fld file. This column was added around 2005 |
| 16 | EXPOSURE | Integer | This column may appear in some combined output files where the combined datasets contained a subset of common objects and therefore exposure times differ for different objects. The column gives the exposure time in seconds for the fibre. |
An Important Note on 2dF Fibre-Pivot Number and 2dfdr Fibre Number
There are two very important, and very different, numbers which one must understand in order to recover the information on which object each fibre was allocated: Fibre slit position AND 2dF Fibre-Pivot position. For the most part there is a one-to-one correspondence between these numbers. Usually the fibre at AAOmega slit position 1 (bottom of the CCD image) will map directly to 2dF Pivot position 1, and 400 will map to 400 (note, 400 is a guide fibre and so maps to a blank space at the top of the CCD image). However, during manufacture or repair of each of the AAOmega slit units, it is sometimes possible for the order of fibres in each of the AAOmega slits to fall out of synchronization with the 2dF Pivot numbering. It is not practical to mechanically alter either position so each of the two fibre numbers (slit position and Pivot position) are propagated in the fibre table.
In the primary image (and also the variance array, stored in the first extension) the fibre at the bottom of the image, which is the fibre at slit position 1, corresponds to the first row in the fibre table (the second fits extension). The table contains a column entry, PIVOT, which gives the 2dF pivot position for this fibre. This is the fibre number seen by the configure software. The very top fibre in a CCD image corresponds to the very last entry in the fibre table (which will be an AAOmega guide fibre in the case of a single AAOmega data set). There is ALWAYS a one-to-one correspondence between each spectrum position in the image and the fibre table. There is typically a one-to-one correspondence between slit position and 2dF pivot position but with a number of known mismatches and discontinuities which are tracked via the PIVOT column of the fibre table.
Examples for Accessing the FITS Fibre Table
This list is not exhaustive.
With configure
One can save a list file (file menu -> ..list) which contains the allocated 2dF Fibre-Pivot number for each allocated fibre. Note, this is the Pivot number for 2dF NOT the fibre number in the reduced 2D spectra file.
2dfinfo
The 2dfinfo procedure comes packaged with 2dfdr. It can be used to recover information on the fibre from the .fits file. The syntax for the command is:
2dfinfo file.fits <option>
If the <option> is omitted then the list of options is given. To recover the fibre table information one would use:
2dfinfo file.fits fibres
IRAF
The IRAF/STSDAS package TABLES has a number of routines designed for manipulating tables. A simple example might be:
IRAF> tdump combined_frame.fits[2] > output.txt
This would create a complete listing of the fibre binary table information and pipe it to an ascii text file. Formatting the output can be achieved with:
IRAF> tprint combined_frame.fits[2] columns="NAME,RA,DEC" > output.txt
IDL
For users of IDL, the NASA IDL astronomy library has some excellent FITS data access routines.
Starting from a combined FITS frame, combined_frame.fits, one might use the following code extracts to manipulate AAOmega data. Note, there are cleverer (and quicker) ways to perform the operations below with the NASA astrolib tasks, the code here is given as a simple example.
file='combined_frame.fits'
;; Read in the spectral image, store the header information
spec=mrdfits(dir+file_comb,0,header0)
;; And the variance array
spec_var=mrdfits(dir+file_comb,1)
;; Make a wavelength vector, note the use of CRPIX1, which is often not expected by many users.
;; If missed, the wavelength solution will tend to be wrong by half a CCD width
crpix=fxpar(header0,'crpix1')-1.0 ; The -1.0 is needed as IDL is ZERO indexed
crval=fxpar(header0,'crval1')
cdelt=fxpar(header0,'cdelt1')
wave=((findgen(n_elements(spec[*,0]))-crpix)*cdelt)+crval
;; Read in the object identification information
fxbopen,unit,file,2
fxbreadm,unit $
,['name','ra','dec','x','y','xerr','yerr','theta','type','pivot','magnitude']$
,id,ra,dec,x,y,xerr,yerr,theta,type,pivot,mag<
fxbclose,unit
;; And read a copy of the sky spectrum subtracted from the data.
;; Note, for a combined frame, this is the sky spectrum from the first file in the list of combined frames.
;; It is a good representative sky spectrum, but should be used with caution for the combined spectral data.
fxbopen,unit,file,7
fxbreadm,unit,['SKY'],sky
fxbclose,unit
Axis Information
Axis information represents the abscissa and ordinate for the image or spectra contained in the file. The abscissa information represents either the pixel number (image) or the wavelength at the centre of each pixel in Angstroms (spectra). The ordinate information represents either pixel number (image) or fibre number (spectra). This is NOT a HDU. Instead the information is held in the FITS standard header keyword values CRVALn, CDELTn and CRPIXn, where n is either 1 (abscissa) or 2 (ordinate). FITS keywords CTYPEn and CUNITn complete the description by holding the axis label and units, respectively. These same keywords are used by external FITS viewers (See fv, ds9, etc.) to describe the axis so 2dfdr output files are correctly handled by these viewers. Within the 2dfdr code, axis information is generally held in a one-dimension vector of 32-bit IEEE floating point values. The vector values are constrained to always being linear since only 3 keywords are used. The transformation from FITS keyword values to vector, and vice versa, is hidden in theTDFIO_AXIS_READ and
TDFIO_AXIS_WRITE routines, respectively.
History: The internal vector format comes from when 2dfdr
used Starlink's Extensible N-Dimensional Data Format (NDF). The NDF
"CENTRE" axis component was used. This was a vector holding the pixel
centre coordinates, with a separate vector for each axis. The NDF axis
attributes of LABEL and UNITS were passed to the FITS keyword string
values CTYPEn and CUNITn, respectively.
N.B. The functionality of this HDU is nearly identical to the WAVELA Extension,
and many of the same issues apply. One of the two should/could
(probably) be eliminated.
Where Used: Every file has this information. Within 2dfdr,
axis information is used when plotting files (read only). The abscissa
values (wavelength) are set during scrunching and splicing. They are
also used to judge wavelengths of interest when matching arc peaks and
known wavelength intensities (read only).
WAVELA
WAVELA is a 2-dimension binary table identical in size to the primary spectral and variance arrays. Each element holds the wavelength in nanometres of the corresponding spectral datum. The data type is 32-bit IEEE floating point. The first dimension is in the spectral direction, whilst the second is in the fibre. This extension is written and read by 2dfdr. Once established the values in this array NEVER change. This information is derived original derived by 2dfdr from keyword values SPECID, GRATPMM, GRATANGR, GRATANGL, CAMANGL and ORDER. The one exception is the original 2df instrument which employs a ray tracing algorithm. Where Used: All types of output ex(tracted) files and arc red(uced) files have an WAVELA HDU. ??? Why ??? It is specifically removed from flat and science red(uced) files. N.B. The ability to provide a separate wavelength for each pixel is NOT used. All values in a single column are identical, and the relationship between row values is constrained to be linear. The one except is the original 2df spectrograph where column differences are found. This format is being retained for future development. N.B. This extension is nearly identical in function to the Axis Information, and many of the same issues apply. One of the two should/could be (probably) eliminated.SHIFTS
SHIFTS is a 2-dimension binary table holding the polynomial coefficients used to rebin (aka scrunch) data onto the calibrated wavelength scale. SHIFTS(FIBNO,COEFF) where- FIBNO is the fibre number, where 1 <= FIBNO <= instrument fibre count
- COEFF is the coefficient number, where 1 <= COEFF <= MAX_SHIFTS (defined in tdfio.inc as 10). That is, there is a set of 10 coefficients for each fibre.
- During the spectra processing for red(uced) arc frames (see
reduce_arc.f). These values are used to rebin fibre flat and science frames. - During spectra processing for red(uced) sky frames (see
reduce_sky.f).These values are used to rebin science frames.
THPUT
A 1-dimension binary table holding the fibre throughput. The vector has one element for each fibre. Each element contains a multiplicative factor to account for differences in fibre throughput. The data type is 32-bit IEEE floating point. The fibre throughput is optionally computed when an object spectra is being reduced using sky fibres in the observation. The results are placed in the object frame's THPUT HDU, and also used to calibrate the object data. This is done when the 'SKYLINE' (or its variants) or 'SKYFLUX' (or its variants) throughput calculation method is chosen. Fibre throughput can also be computed during the reduction of a sky frame. The values are placed in the sky frame's THPUT HDU, and can later be used to calibrate object frames. This is done when the 'OFFSKY' throughput calculation method is chosen when reducing the object frame. Object spectra is scaled for fibre differences by dividing by the throughput. Where Used: This extension is written and read by 2dfdr.SKY
A 1-dimension binary table holding the combined and normalised sky spectrum. The vector has one element for each wavelength. The data type is 32-bit IEEE floating point. The sky spectrum is optionally computed when an object spectra is being reduced. When it is, the sky spectrum is subtracted from each object spectrum, and the sky spectrum values are stored in this extension. Each entry contains the 'typical' sky spectrum used in the data reduction; 'typical' because for a combined frame it is not obvious how the final sky spectrum for each fibre should be represented. The values in the SKY extension are used during splicing of AAOmega blue and red spectra for what purpose???. Note: The variance information is correctly propagated, the sky spectrum is not presented here for this purpose. Where Used: This extension is written and read by 2dfdr.TELCOR
A 1-dimension binary table holding the telluric absorption correction used. The vector has one element for each wavelength. The data type is 32-bit IEEE floating point. Where Used: This extension is ONLY written by 2dfdr.SIGMAPRF
A 2-dimension binary array which appears ONLY in the tramline map file. The data type is 32-bit IEEE floating point. Under the assumption that the tramline PSF is a Gaussian function, the sigma value is the estimate of the sigma of the Gaussian. The array has a sigma for each fibre and each spectra pixel. This is currently (Nov2010) being explored by the astronomers with various specific experimental data. Once further confidence is obtained in the derived values they will be implemented instead of the fixed value in the various coding sections including optimal extraction. Where Used: This HDU is currently ONLY written by 2dfdr.DELTA
N.B. This HDU is obsolete. The Fortran code to produce this HDU has NOT been converted from NDF to FITS. FIT extraction does NOT work because in also has not been converted to FITS. See bug report History: A 2-dimension binary table identical in size to the IMAGE array holding ???. The data type is 32-bit IEEE floating point. This and the SIGMA2 HDU were used for FIT spectra extraction. Both extensions are created when the tramline map is made. The arrays hold ??? sigma and delta of the Gaussian fit to spectra profiles ??? <<-- FILL IN HERE! They are used during the spectra extraction but again only when FIT extraction is requested. This extension is written and read by 2dfdr. Where Used: No recent 2dfdr files have this HDU.NDF_CLASS
A 1 x 1 binary table. Its value was part of the implementation of OO (object-orientated) Fortran. Its use is deprecated. A 10-character string describing the file type. Known types areMFFFF |
Fibre flat field image, raw and im(age) |
MFSFFF |
Fibre flat field spectra, ex(tracted) and red(uced) |
MFARC |
Arc image, raw and im(age) |
MFSARC |
Arc spectra, ex(tracted) and red(uced) |
MFOBJECT |
Science image, raw and im(age) |
MFSOBJECT |
Science spectra, ex(tracted), red(uced), combined and spliced |
MFFLX |
Flux calibration image, raw and im(age) |
MFSFLX |
Flux calibration spectra, ex(tracted) and red(uced) |
BIAS |
Bias frame image, raw and red(uced) |
LFLAT |
Long-slit flat image, raw and red(uced) |
REDUCED
A 1 x 1 binary table indicating the file was reduced. If present it contains the single logical value "true". Its use is deprecated. Where Used: All red(uced) 2dfdr output files contain this HDU.Sarah Brough (sb_at_aao.gov.au)
Ron Heald (rwh_at_aao.gov.au)