IDFS data sets can be as simple or complex as you want. We recommend the simple, however. This document will show you the steps required to make an IDFS file from raw telemetry and offer some case studies which will hopefully make your job easier.
We will also describe a library of routines written to facilitate the writing of IDFS's. This document assumes you are a programmer with some ability. For the purpose of this document, we use C++ and PERL. The C++ used is VERY minimal. Anyone with a knowledge of C should be able to grasp the C++ calls. You will need a C++ compiler to use the supplied routines, though. You can actually use any language you want to create IDFS files.
Lastly, we assume a knowledge of SDDAS. SDDAS is not necessary for IDFS, but it will make things much easier. I encourage you to read the Overview of the Data System and the Overview of the Analysis System if you are not familiar with SDDAS.
You will need the following:
The first step to IDFS creation is deciding how your hierarchy is going to look. Typically, this is the easiest part. If you are adding to a project, there is no problem. If you are starting from scratch, just pick logical project, mission, experiment, and instrument names from the technical manuals. The virtual instruments are a bit different. All instrument data is broken down into virtual instruments depending on a variety of things which are completely arbitrary. It does not matter from a software point of view how the virtuals are broken out, but the IDFS format itself places some restrictions on the virtuals.
First, each sensor within a virtual must have common number of samples and other fields which are within the VIDF file. Most timing information ought to be the same for every sensor within a virtual as well. Realistically, you ought to make sure that each of the sensors within the virtual are related in some way.
Now it is time to write the processing software! Basically, you need to write some code which will put the data you have into the IDFS data format. Luckily, we have written a library for you which may simplify the task a bit.
The VIDF is the hardest part of creation of IDFS; if you can understand that, you have most of the problems resolved.
Writing a VIDF is not easy and it takes some skill to create a working VIDF. Typically, trial and error is the best teacher though looking at other VIDF's will also help.
Get the VIDF documentation and follow it line by line creating a text file with your favorite text editor. Take care to not make any errors and when you are done, you can try compiling it into binary format. To do that, use the SDDAS supplied program mk_idf sending it the name of the text file and the name of the binary file to be created. Your text file is typically referred to as the "V file".
We have also another way of creating a VIDF file. This is with our GUI VIDF editor. Read about it in the VIDF Editor documentation. With the editor, a wizard is available to guide you through the creation of a VIDF.
VIDF's have special names. The name is built up of the following:
For example, a virtual instrument named ABCD that is valid from 1972, day 103, 11:04 am is named ABCD19721031104V. After the VIDF file is compiled, the V is replaced with an I and is referred to as the "I file".
If mk_idf does not core dump, the VIDF might very well be valid. However, there is one final check. There is a program called rd_idf (included with SDDAS) which will read the binary version. Send it the name of the I file on the command line. The rd_idf output should look almost identical to what you put into the V file.
Creating a PIDF is actually a very easy task. The PIDF gives the plot programs common answers to questions as well as coordinating the VIDF with the analysis of the data. First, collect all the information on units and ranges of the units and the sensor names (as in the VIDF) as well as any other information which was needed to create the VIDF.
Next, load up the PIDF Editor from the "Control Panel->Editors->PIDF Editor". The PIDF Editor is a GUI tool useful for creation of the PIDF. Between this program and a text editor, you can very rapidly create PIDF files. Since this process is fully GUI driven and fully documented, I will not get into details here. When you completed creating a PIDF, check your PIDF file with dump_pidf.
I have some other recommendations for PIDFs. First, make most of the units default to "Log" scaling. Next, always make a "Unitless" units value. It will be necessary for the Data Quality portion. Keep your units labels small. While filling out the Sensor portion, "Name" will be shown to the user on the menus while "Component" will be shown on the plot. You have more room on the menus than the plot. Label them accordingly.
Useful links for PIDF creation:
This is another relatively easy step. To get added into SDDAS, you need to add the hierarchy to the config files, put the data into the $SDDAS_DATA area in the correct location, and build the databases. Lets take these steps one at a time:
The two config files which need to be edited are in $SDDAS_HOME/config. They are DB.cfg and PROJECTS.cfg. DB.cfg needs editing if you are adding to the hierarchy at the Projects level. If you are only adding onto an already existing project, this file will not change.
The PROJECTS.cfg contains a list of all the Projects, Missions, Experiments, Instruments, and Virtual Instruments. Basically, find the correct location of what you need to add to the hierarchy and add it in.
There are programs to make this task easier. In the "Control Panel-> Administration Tools-> Update Projects Config (or Update DB Config)."
The directory in which the data goes corresponds directly to the hierarchy name. The correct directory is named as follows:
$SDDAS_DATA/<project>/<mission>/<experiment>/<instrument>
SDDAS_DATA is an environment variable which should be set by sdconfig, but can be anything. This is the base SDDAS data directory. After that, the project, mission, experiment, and instrument are all exactly the same as what you selected for the VIDF/PIDF. What needs to go in that directory is the VIDF, the PIDF, and the header and data files.
Databases can be generated any way possible, but currently we have several scripts and programs which are used to help the IDFS/SDDAS programmer generate databases. After you put all the data in the correct location, the GenDbAsc.pl script can help you create the database entries. You may want to check the database entries to be sure they are OK. There should be no negative numbers or any odd values in there. Once you have the entries, you can create the databases.
Databases are kept in a directory called Database under the experiment level. In other words, there is a directory called:
$SDDAS_DATA/<project>/<mission>/<experiment>/<Database>
Move the hd.asc and the i.asc files created by GenDbAsc.pl into this directory and use the program "UpdateDb <instrument>". After the databases are created (look for the <instrument>.HD.DBF files), run the Index <instrument>". After this, the database files are ready for use!
Useful links for adding to SDDAS/Database Creation:
Data validation is simply plotting the data to see if it looks correct. The typical program for this is SpectroScalar although any program will work. Basically, just plot the data as you would normally and see that correct values show in the plot and that you are seeing what you would expect to see. If you are not, then obviously something is wrong with the data. Common things to check are the sen_mode variable in the VIDF, the timing information in your processing software, and the values within the data record.
We have a program called dump_idfs which will basically dump out numbers. Use it to dump the numbers from the data file. It is not pretty, but it should give you an idea if the number are correct or marginally out of range.
Another useful tool is "od" which is included with every Unix system (try "man od"). This tool will do an octal dump of the header or data file. Since the data file is generally long integers, use "od -D <data file name>" to view it. The numbers should look familiar. For the header file, most of the values are short integers, so use use "od -d <data file name>" to view them. Once again, some of the number should look familiar and you should be able to read the file without problems.
After everything is correct (keep making changes to the processing code, VIDF, PIDF, and databases until it is correct!), you are ready to archive the data. All archiving means the data is ready to made available to the rest of the world and is able to be promoted from your site.
You must first be an archive site. If you are not an archive site, read up on becoming on archive site.
You may also send the data here to be archived, email me for details.
You will no doubt have many problems converting your data to IDFS. Here are some of the ones I had and some solutions to the problems. Email me if you find some other problems which others might have and their solutions.
Solution: error in VIDF - check it with rd_idf and make sure it says what you think it should say.
Solution: error in data length size in VIDF - fix it.
Solution: type cast all elements going in to the biggest one or split them out into seperate virtuals.
Solution: data latency is incorrect in VIDF file or the header file. This will also show through in the data base (file will have incorrect end time for instance). Correct the data latency (remember: its in microseconds).
Solution: There is a problem in the processing software. Check the
following things:
Solution:Either there is a problem with the processing software or the PIDF file is pointing to the same VIDF number for every sensor. They need to be different!