SolarSoft techniques for dealing with 3D data sets

Working with 3D data always merits some thought. Some general guidelines, hints and options are reviewed here. The actual approach taken is sensitive to local system resource contraints (memory) as well as the actual application and acceptable tradeoffs between spatial resolution, time resolution/cadence, time range etcetera. Some of the information presented here is repeated in the SSW concepts document. Although the examples may refer to a specific instrument such as TRACE, the concepts are generally applicable to any 3D collection of large images within the SolarSoft IDL environment.

As discussed in the SSW concepts document, most of the read routines accept a vector of file names with an arbitrary number of elements. For files which are internally 3D, there may be an additional SS (SubScripts) parameter. The bottom line is that a user can always specify a 3D data read which will overload the local system. On a more positive note, the read routines (and FITS conversion routines) are designed to permit reading of an extremely large number of headers into an index vector without reading any data. This index vector can be used to sub select (pre filter) the images of actual interest and the used to perform the actual data read. The following 3 element dummy sequence is fundamental:.

• IDL> read_xxx, filelist, -1, index ; Read the headers from filelist into an ->index vector
• IDL> ss=[filters(index)] ; where, sswhere, trace_sswhere, struct_where, etc
• IDL> read_xxx, filelist, ss, index, DATA ; now read DATA for filtered subset (ONLY)

Prefiltering on time

Constraining the time range of interest is an obvious prefilter for most applications. Often the file finding routines like xxx_files offer a first cut time range filter and the contents of your filelist implies a constrained time range.

Prefiltering on image parameters

Filtering on wavelength, field of view, exposure duration, and other instrument parameters is accomplished either by an IDL where statement or via a utility widget (sswhere, trace_sswhere, struct_where...)
One more time: ...read headers -> filter headers -> read filtered data...

IDL> read_trace, tracef, -1, tindex ; Read all the image headers in 'tracef' -> tindex

IDL> ss=trace_sswhere(tindex) ; subselect desired instrument parameters from the WIDGET

IDL> read_trace, tracef, ss, tindex, tdata ; read the filtered subset of data

Trading off cadence for time range coverage

The routine grid_data.pro acts upon any index vector to allow sub sampling a cube at a quasi fixed cadence. This is often applied after other instrument prefilters to reduce the number images over a given time range. The routine tim2dset.pro is useful for subselecting a high cadence index using image times of a lower cadence instrument - ie, identify the ss subset of high time resolution images closest in time to the lower time resolution data set (or discrete event times)

For example, assuming the ss vector implies too many images after image parameter filtering and a time resolution of 30 minutes is sufficient for this applicataion - then:

IDL> sss=grid_data(index(ss),min=30)) ; subselect one every 30 minutes
IDL> read_trace,files,ss(sss),index,data ; match parameters and reduced cadence
NOTE that the new (sss=reduced cadence) subscripts are derived relative to the original prefiltered subscripts ss so the read_trace call uses ss(sss) for image numbers

Assume index0 represents high cadence image times from instrument xxx and index1 are the records from a lower cadence instrument yyy (and assuming some time overlap between the two index vectors). Then the subscripts for instrument xxx closest to yyy images are derived via:

IDL> sss=tim2dset(index0(ss), index1, delta=seconds) ; match index (time) vectors
IDL> read_xxx, files, ss(sss), indexx, datax ; now read the xxx images closest in time to images yyy
NOTE: that the new subscripts sss are relative to some previous filtered subset ss so the read_xxx call uses ss(sss) nomenclature.

Trading off time range coverage for cadence/time resolution

If you want the highest time resolution, you can always subselect a smaller time range - this can be done within xxx_sswhere widget, sel_timrange, or just via array subscripting of the ssvector. Some examples:

IDL> read_trace,files,ss(0:9),index,data ; read first 10 'prefiltered' images

IDL> read_trace,files,last_nelem(ss,10),index,data ; read last 10 'filtered' images

IDL> ss=sel_timrange(index,'1-jun-1998 00:10','1-jun-1998 00:15') ; define ss within time range
IDL> read_trace,files,ss,index,data ; now read associated index,data

Trading off spatial resolution

Many read routines (including read_trace, read_eit, mreadfits) offer the keyword parameter OUTSIZE to resize the data cube during the read operation. Here is a TRACE example (assume ss already contains the prefiltered subscripts for a given filter and field of view combination)

IDL> read_trace, files, SS, index, data ,outsize=386 ; read the data; rebin on the fly -> 386x386xNIMAGES

IDL> mreadfits_fixup, index, data ; fix INDEX parameters to match rebinned data

IDL> trace_prep,index,data,oindex,odata ; prep the data (L0->L1)

IDL> sdata=bytscl(alog10(odata>1)) ; scale the data

IDL> xstepper, sdata, get_infox(oindex,'date_obs,wave_len') ; display the data (3D cube reviewer)

Avoiding large memory operations Note that the xxx_prep routines (sxt, trace, eit) permit use of input and output in terms of filenames in addition to index,data which can avoid large memory operations. Usually most useful processing large number of files in a non interactive session.