NASA Logo, National Aeronautics and Space Administration
STEREO / WAVES

STEREO-WAVES Data Processing

Timely Data Processing

Since one of the major goals of the STEREO program is Space Weather predictions, it will be necessary for the SWAVES team to process the data very quickly. Our experience with the Wind/WAVES data processing has shown that within a few hours of receipt of the data (and sometimes within a few minutes), we can issue a Space Weather prediction of the arrival time of a CME-driven shock at Earth. Assuming STEREO data is available to investigators within 24 hours of receipt, this prediction is then 2-3 days in advance of impact. With SWAVES, the predictions will be more sophisticated but we expect to be able to issue them very quickly, within hours of our receipt of level zero data.

The SWlib concept

The fundamental building block of the data analysis system is a programmer interface layer that simplifies and makes consistent the user’s ability to acquire and manipulate telemetry data, calibrated data, ancillary data such as time, spacecraft coordinates, and orientation, and data from other sources –including other spacecraft missions, current and future. This data access architecture was successfully developed and implemented for Wind/WAVES, and the system for SWAVES – to be known as SWlib – will inherit most elements of the overall design, supplemented by modern access methods particularly with respect to Web availability and GUI presentation.

In this architecture, a database of item names, where each item refers to a specific piece or set of information is the backbone of the system. Items can include status flags, times, raw telemetry values, calibration tables, or entire files of constants. Items can also refer to processed data, e.g. values in physical units, derived from the invocation of a function against incoming data. Calibration is performed on demand when the calibrated item is requested – thus ensuring that any access to the data is accompanied by the most up-to-date calibration, and eliminating the need for reprocessing.

The database entry for each item shows its origin (bit field within telemetry, file, constant etc.) and how it is to be extracted and presented to the user. For telemetry, for instance, the packet location and the pattern of how that element of data is laid out in the packet is described, whereas for items coming from a file (e.g. a calibration table) the name of the file and information about how to read the contents is provided. Each item is also given a validity in time – so that items generated early in the life of the experiment (on the bench, for example) and that are no longer included in the flight version, are still in the data base, and may still be used with the bench data to which they belong – but are invalid if an attempt to invoke them is made, for example, a year into flight. If an item changes its source as time goes on – perhaps in a different part of the packet or coming from a file instead of a single constant - the database timestamp permits the use of the correct version of the item based on the time of its validity in each of its incarnations. This permits the evolution of bench/flight software before and after launch with no code changes or versioning problems in the ground software. Any changes in the telemetry configuration or content are made by adding to or amending items in the database, without any changes to code itself.

This strategy was invoked early in the development phase for Wind/WAVES and the software developed at that time and based on WINDlib has been in use since that time. The WINDlib software is a mature product and a well-developed architecture that can easily be ported to the STEREO framework.

For the most part, products are generated as they are required and there is no extra ‘processing’ step generating mid-level products which must be redone when calibrations are improved (see Data Archival below). This trades some small extra time in accessing data for the assurance that calibrated data obtained are up-to-date. This trade, even with the larger data set and slower computers of the Wind/WAVES era, has proved to be viable, and the faster computers to be anticipated 5 years hence make it an even more attractive option. To assist in scanning the data for events of interest, however, we will produce daily summary plots similar to those produced for Wind/WAVES that show an overview of the days’ activity across all the sampled frequencies, and which also confirm the status of the instrument and its components. This type of plot, which will be quickly available to the public in PostScript and Web-based formats, has proven very useful in identifying solar events, isolating data handling problems, and identifying instrument anomalies. We will also produce a set of reduced processed data giving users fast access to one minute averages. We also expect the STEREO science team to adopt an ISTP-style key parameter database and will generate data products for public access.

Calibration

Individual science team members have responsibility for determining accurate calibrations for data from each instrument, and for providing appropriate algorithms and tables of constants to the SWlib manager at UMN. These calibrations will be included in SWlib and tested before release. Calibrated data acquires item names separate from, but related to, the name of the original telemetry item from which they are derived, e.g. Z_PRIME is telemetry, and Z_PRIME_LOGSFU is calibrated in units of Log10 (Solar Flux Unit).

Data Storage

We anticipate that all data will be available online at each institution for the duration of the mission, with regular backups to offline storage for security. The relatively small size of the original data set (approximately 10MBytes per day), the lack of a proliferation of intermediate products, and the increasing capacity of disk storage makes this a feasible strategy. The benefits include fast access for users, especially for studies encompassing long periods of data. Data will also be available on DVD for long term and off-line use, taking advantage of the production capability of the SECCHI experiment.

Data Access

SWlib forms a Programmer Interface (API) to the experiment data and is appropriate for individuals having a sophisticated knowledge of the experiment and its data. Experience on Wind/WAVES shows that a more intuitive, web-based layer containing tools is an appropriate addition to the access choices available to the end-user of the data. We envisage creating an interactive environment, accessible through the web, that permits users to generate plots, subset data, create averaged data sets, and correlate selected quantities with others in a number of different ways. Requirements for the exact capabilities to be available will be generated through extensive consultation with the user community, and prototype interfaces will be designed to iterate these requirements to a useful product early in the development lifecycle.

Data Archival

We expect to provide the NSSDC with selected parameters and overview plots. It is likely the STEREO science team will, in consultation with the broader community and the NSSDC, determine to produce a number of public data products along the lines of those produced by the ISTP program. The SWAVES team will provide software for the generation of such key parameters and tools for their use. As part of our process during our daily receipt of data, we will produce and pipeline to the NSSDC averaged data sets of key parameters in physical units in a form that will be useful to other researchers. If there is significant change in calibration, these products will be re-created with the new calibration sets and resent to NSSDC. Final archival of the SWAVES data will be accomplished by providing the original raw data set and the interface tools to NSSDC, with documentation sufficient to permit the use of the data. All data will be delivered to NSSDC within the required three-month interval from initial receipt, probably much sooner based on our Wind/WAVES experience.

Documentation

On-line documentation in the form of help files, user guides, programmer’s guides and instrument descriptions will be provided. The database of items contains very specific and detailed information about the contents of the telemetry and the calibration of the instrument and is an essential ‘handbook’ to the experiment as a whole. The database also documents the flag status of all instrument flags and makes them directly accessible to the user.

A utility will be provided that generates from the database a listing of the contents and layout of any particular packet, as well as showing what quantities are available for use by the programmer for each instrument subsystem. Similar utilities were provided for WINDlib, and the design will be portable to the new environment.

NASA Logo - nasa.gov