Document No. 92048PRO0114
Date 17 February 1992
Author M A Hapgood
Title Proposed structure of UKS databases
Project AMPTE DATABASE REGENERATION
Organisation SPACE SCIENCE DEPARTMENT
RUTHERFORD APPLETON LABORATORY
SCIENCE AND ENGINEERING RESEARCH COUNCIL
I have constructed trial databases under R-EXEC for each AMPTE-UKS dataset as shown in the following sections.
As far as it is practical, I have defined the time fields to be identical with those in the new IRM databases. Thus it will be straightforward to join UKS and IRM datasets on the time fields for comparative studies. The use of the R-EXEC JOIN function requires that the common fields have identical names, types, dimensions and null values. I have used names for time fields as shown in the list below.
UTY = year (two digits)
UTD = day of year
UTH = UT hour
TrueSeconds = seconds in hour as a floating point number
Seconds = seconds in hour rounded to nearest multiple of 5
All time fields have dimension 1 and zero null values. UTY, UTD and UTH and Seconds are type S (two byte integers) while TrueSeconds is type R (real). The Seconds field is present in all UKS and IRM data but the TrueSeconds field is only present in UKS ion and wave data files.
The Seconds field in the UKS databases is designed specifically so that naive use of the R-EXEC JOIN function between IRM and UKS databases will combine records with the same time and thus give a scientifically meaningful result. In the present case JOIN will combine records on the common fields UTY, UTD, UTH and Seconds. To ensure that JOIN works reliably, these are all defined as integer fields and the Seconds field is rounded to a multiple of 5 in both IRM and UKS data. Care has been taken to ensure that all other fields in the UKS database have names that are distinct from fieldnames in the IRM database.
Note also that the R-EXEC JOIN function requires both input files to be ordered on the common fields. This is to be expected for time fields.
To ensure that all information is available to the database user, the TrueSeconds field contains the precise start time of each integration. This may be required for the study of rapid changes on the order of the integration time.
Each UKS database contains a one character validation field. It always follows immediately after the time fields and has a name of the form STRvalid, where STR is the name of the datastream, namely one of MAG, FTR, SW, EL, AC, SFA or POW. The values of this field are:
* = unvalidated (default)
blank = validated (i.e. '*' removed)
X = requires re-processing
B = cannot re-process but bad data
N = may contain noise or outliers
S = systematic instrumental effects
M = quality flags provided by PI (ion data only)
Other values may be defined if needed. The field has a null value '*'.
The processing required to generate each database from the source data involves two steps:
The key files used to perform these operations are listed in each section and are on RLVK unless otherwise indicated. A command file allows both steps to be done in a single batch job. To process different datasets, dates should be changed by editing the appropriate COM file.
The R-files produced by the loading procedure are placed in a directory referenced by the logical name WORK[. The form of file name is UKSyr_ddd_str.REX, where:
yy = year, two digits i.e. 84 or 85.
ddd = day of year, three digits including leading zeros.
str = datastream name, two or three characters e.g. MAG.
The procedures described in the following sections generate seven separate database files: 1 magnetic field, 2 ions (FTR and SW), 1 electron, and 3 wave (AC, SFA and power). Can these files be combined? I think the answer is no as, by and large, the datasets have different integration periods:
Thus it will be better to have separate files for each of the seven datasets and provide facilities for users to join datasets together when needed. The naive use of JOIN will produce meaningful results because of the definition of the Seconds field described previously. More sophisticated joins are possible through manipulation of the TrueSeconds field. Different types of joins will have different effects on the statistical homogeneity of the resulting dataset.
This approach is consistent with the thinking which has emerged from ESIS: that there are several ways of joining datasets when that join is made on a continuous variable such as time. Thus databases should be constructed in a way which allows users to choose a suitable strategy for joining datasets. The strategy chosen may be supplied by the database or one provided by the user. The choice of strategy will be determined by scientific objectives.
The contents of the trial database are taken from the magnetic field data files supplied to Chunkey by Malcolm Dunlop and Chris Mier, and now on-line in the GDF. All files are in directory DISK[PLASMA2:[MAH.DATABASE.MAG] unless otherwise indicated.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 Seconds 1 S 6 - 0 2 MAGvalid 1 C 1 - * 1 B1 1 R 7 2 -413262.31250 4 B2 1 R 7 2 -413262.31250 4 B3 1 R 7 2 -413262.31250 4 MagB 1 R 7 2 -413262.31250 4where:
B1 = Magnetic field GSE X component in nanoteslas.
B2 = Magnetic field GSE Y component in nanoteslas.
B3 = Magnetic field GSE Z component in nanoteslas.
MagB = Average magnetic field magnitude in nanoteslas.
Note that MagB**2-B1**2-B2**2-B3**2 is a measure of the variance.
The two different types of ion data are treated separately as shown below. In each case the contents of the trial databases are taken from the ion data files produced at RAL by running Rob Gowen's AMPTE Reprocessing software. All files are in directory DISK[PLASMA2:[MAH.DATABASE.IONS] unless otherwise indicated.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 TrueSeconds 1 R 8 3 0.00000 4 Seconds 1 S 6 - 0.00000 2 FTRvalid 1 C 1 - * 1 Iden 1 R 13 5 -413262.31250 4 Ivel 3 R 13 5 -413262.31250 12 Itemp 1 R 13 5 -413262.31250 4 Icounts 1 R 8 1 0.00000 4 Iquality 1 S 3 - 0 2where:
Iden = ion density per cm**-3
Ivel = X, Y and Z components of ion bulk velocity in km sec **-1
Itemp = ion temperature in eV
Icounts = ion total counts in distribution
Iquality = ion data quality indicator
Note. The term hhmm in the input data file name is the UT start time of the data in hours and minutes.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 TrueSeconds 1 R 8 3 0.00000 4 Seconds 1 S 6 - 0.00000 2 SWvalid 1 C 1 - * 1 PRden 1 R 13 5 -413262.31250 4 PRvel 3 R 13 5 -413262.31250 12 PRtemp 1 R 13 5 -413262.31250 4 PRcounts 1 R 11 5 0.00000 4 SWquality 1 S 4 - 0 2 ALden 1 R 13 5 -413262.31250 4 ALvel 3 R 13 5 -413262.31250 12 ALtemp 1 R 13 5 -413262.31250 4 ALcounts 1 R 11 3 0.00000 4where:
PRden = proton density per cm**-3
PRvel = X, Y and Z components of proton bulk velocity in km sec **-1
PRtemp = proton temperature in eV
PRcounts = proton total counts in distribution
SWquality = proton/alpha particle data quality indicator
ALden = alpha particle density per cm**-3
ALvel = X, Y and Z components of alpha particle bulk velocity in km sec **-1
ALtemp = alpha particle temperature in eV
ALcounts = alpha particle total counts in distribution
Note. The term hhmm in the input data file name is the UT start time of the data in hours and minutes.
The contents of the trial database are taken from the electron data file produced by Chunkey's program. All files are in directory DISK[PLASMA2:[MAH.DATABASE.ELECTRONS] unless otherwise indicated.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 Seconds 1 S 6 - 0 2 ELvalid 1 C 1 - * 1 ElecDen 1 R 13 5 -413262.31250 4 ElecFlux 1 R 13 5 -413262.31250 4 ElecHflux 1 R 13 5 -413262.31250 4 ElecPpar 1 R 13 5 -413262.31250 4 ElecPper 1 R 13 5 -413262.31250 4 NEbins 1 B 2 - 0 1 Energy 48 R 11 3 0.00000 192 Vsden 48 R 11 3 0.00000 192where:
ElecDen = density in number/(metre to power 3)
ElecFlux = number flux in number/second/metre to power 2)
ElecHflux = heat flux in Watts/(metre to power 2)
ElecPpar = parallel pressure in Newtons/(metre to power 2) i.e.
Pascals.
ElecPper = perpendicular pressure for one degree of freedom in
Pascals. Note that total pressure = 2 * ElecPper + ElecPpar.
NEbins = number of energy bins used.
Energy = array of 48 energy levels measured in eV, note that in
order to cope with the fact that the number of energy levels varies
with instrument mode, 48 values are always written out but only
values for 1 to number of energy bins used are meaningful.
Vsden = array of 48 velocity-space densities given in number
(seconds to power 3)/(km to power 6). They are computed for the
"centre" angle bin (j_cent=jbins/2), where jbins is the total number
of angular bins. It should correspond to near 90 degrees.
The three different types of wave data are treated separately as shown below. In each case the contents of the trial database are taken from wave data files supplied by Sheffield and were then processed as described below. All files are in directory DISK[PLASMA2:[MAH.DATABASE.WAVES] unless otherwise indicated.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 TrueSeconds 1 R 8 3 0.00000 4 Seconds 1 S 6 - 0.00000 2 ACvalid 1 C 1 - * 1 ACspec 52 R 13 5 -413262.31250 208where:
ACspec = power spectral density in Volts**2 metres**-2 Hertz**-1 at
52 frequencies in the range 0.11 to 3.9 kHz. A mapping of stepnumber
to frequency is AC_FREQ.REX in the same directory.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 TrueSeconds 1 R 8 3 0.00000 4 Seconds 1 S 6 - 0.00000 2 SFAvalid 1 C 1 - * 1 SFAspec 256 R 13 5 -413262.31250 1024where:
SFAspec = power spectral density in Volts**2 metres**-2 at 256
frequencies in the range 0.392 to 132 kHz. A mapping of stepnumber to
frequency is SFA_FREQ.REX in the same directory.
Name Dim Type Width Dec Null Bytes ------------ ------ ---- ------ --- --------------- ------ UTY 1 S 2 - 0 2 UTD 1 S 3 - 0 2 UTH 1 S 2 - 0 2 TrueSeconds 1 R 8 3 0.00000 4 Seconds 1 S 6 - 0.00000 2 POWvalid 1 C 1 - * 1 AvPower 1 R 17 10 -413262.31250 4 PkPower 1 R 17 10 -413262.31250 4where:
AvPower = average wave power during integration period in Volts**2
metres**-2. Measured over 0.11 to 3.9 kHz.
PkPower = peak wave power during integration period in Volts**2
metres**-2. Measured over 0.11 to 3.9 kHz.
WDC Introduction Contact Staff General Help Index Previous
Produced by World Data Centre for STP