Working with XArray
There are two functions for working with XArray Datasets, one for converting
a CDF to a DataSet, and one for going the other way. To use these you need
the xarray
package installed.
These will attempt to determine any ISTP Compliance, and incorporate that into the output.
- cdflib.cdf_to_xarray(filename, to_datetime=False, to_unixtime=False, fillval_to_nan=False)[source]
This function converts CDF files into XArray Dataset Objects.
- Parameters
filename (str) – The path to the CDF file to read
to_datetime (bool, optional) – Whether or not to convert CDF_EPOCH/EPOCH_16/TT2000 to datetime, or leave them as is
to_unixtime (bool, optional) – Whether or not to convert CDF_EPOCH/EPOCH_16/TT2000 to unixtime, or leave them as is
fillval_to_nan (bool, optional) – If True, any data values that match the FILLVAL attribute for a variable will be set to NaN
- Returns
An XArray Dataset Object
- Example MMS:
>>> #Import necessary libraries >>> import cdflib >>> import xarray as xr >>> import os >>> import urllib.request
>>> #Download a CDF file >>> fname = 'mms2_fgm_srvy_l2_20160809_v4.47.0.cdf' >>> url = ("https://lasp.colorado.edu/maven/sdc/public/data/sdc/web/cdflib_testing/mms2_fgm_srvy_l2_20160809_v4.47.0.cdf") >>> if not os.path.exists(fname): >>> urllib.request.urlretrieve(url, fname)
>>> #Load in and display the CDF file >>> mms_data = cdflib.cdf_to_xarray("mms2_fgm_srvy_l2_20160809_v4.47.0.cdf", to_unixtime=True, fillval_to_nan=True) >>> print(mms_data)
>>> # Show off XArray functionality >>> >>> # Slice the data using built in XArray functions >>> mms_data2 = mms_data.isel(dim0=0) >>> # Plot the sliced data using built in XArray functions >>> mms_data2['mms2_fgm_b_gse_srvy_l2'].plot() >>> # Zoom in on the slices data in time using built in XArray functions >>> mms_data3 = mms_data2.isel(Epoch=slice(716000,717000)) >>> # Plot the zoomed in sliced data using built in XArray functionality >>> mms_data3['mms2_fgm_b_gse_srvy_l2'].plot()
- Example THEMIS:
>>> #Import necessary libraries >>> import cdflib >>> import xarray as xr >>> import os >>> import urllib.request
>>> #Download a CDF file >>> fname = 'thg_l2_mag_amd_20070323_v01.cdf' >>> url = ("https://lasp.colorado.edu/maven/sdc/public/data/sdc/web/cdflib_testing/thg_l2_mag_amd_20070323_v01.cdf") >>> if not os.path.exists(fname): >>> urllib.request.urlretrieve(url, fname)
>>> #Load in and display the CDF file >>> thg_data = cdflib.cdf_to_xarray(fname, to_unixtime=True, fillval_to_nan=True) >>> print(thg_data)
- Processing Steps:
- For each variable in the CDF file
- Determine the name of the dimension that spans the data “records”
Check if the variable itself might be a dimension
The DEPEND_0 likely points to the approrpiate dimensions
If neither of the above, we create a new dimensions named “recordX”
- Determine the name of the other dimensions of the variable, if they exist
Check if the variable name itself might be a dimension
The DEPEND_X probably points to the appropriate dimensions for that variable, so we check those
If either of the above are time varying, the code appends “_dim” to the end of the name
If no dimensions are found through the above checks, create a dumension named “dimX”
Gather all attributes that belong to the variable
Add a few attributes that enable better plotting with built-in xarray functions (name, units, etc)
Optionally, convert FILLVALs to NaNs in the data
Optionally, convert CDF_EPOCH/EPOCH16/TT2000 variables to unixtime or datetime
Create an XArray Variable object using the dimensions determined in steps 1 and 2, the attributes from steps 3 and 4, and then the variable data
Gather all the Variable objects created in the first step, and separate them into data variables or coordinate variables
Gather all global scope attributes in the CDF file
Create an XArray Dataset objects with the data variables, coordinate variables, and global attributes.
- cdflib.xarray_to_cdf(xarray_dataset, file_name, from_unixtime=False, from_datetime=False, istp=True, record_dimensions=[], compression=0)[source]
This function converts XArray Dataset objects into CDF files.
- Parameters
xarray_dataset (xarray.Dataset) – The XArray Dataset object that you’d like to convert into a CDF file
file_name (str) – The path to the place the newly created CDF file
to_datetime (bool, optional) – Whether or not to convert variables named “epoch” or “epoch_X” to CDF_TT2000 from datetime objects
to_unixtime (bool, optional) – Whether or not to convert variables named “epoch” or “epoch_X” to CDF_TT2000 from unixtime
istp (bool, optional) – Whether or not to do checks on the Dataset object to attempt to enforce CDF compliance
record_dimensions (list of str, optional) – If the code cannot determine which dimensions should be made into CDF records, you may provide a list of them here
compression (int, optional) – The level of compression to gzip the data in the variables. Default is no compression, standard is 6.
- Returns
None, but generates a CDF file
- Example CDF file from scratch:
>>> # Import the needed libraries >>> import cdflib >>> import xarray as xr >>> import os >>> import urllib.request
>>> # Create some fake data >>> var_data = [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> var_dims = ['epoch', 'direction'] >>> data = xr.Variable(var_dims, var_data)
>>> # Create fake epoch data >>> epoch_data = [1, 2, 3] >>> epoch_dims = ['epoch'] >>> epoch = xr.Variable(epoch_dims, epoch_data)
>>> # Combine the two into an xarray Dataset and export as CDF (this will print out many ISTP warnings) >>> ds = xr.Dataset(data_vars={'data': data, 'epoch': epoch}) >>> cdflib.xarray_to_cdf(ds, 'hello.cdf')
>>> # Add some global attributes >>> global_attributes = {'Project': 'Hail Mary', >>> 'Source_name': 'Thin Air', >>> 'Discipline': 'None', >>> 'Data_type': 'counts', >>> 'Descriptor': 'Midichlorians in unicorn blood', >>> 'Data_version': '3.14', >>> 'Logical_file_id': 'SEVENTEEN', >>> 'PI_name': 'Darth Vader', >>> 'PI_affiliation': 'Dark Side', >>> 'TEXT': 'AHHHHH', >>> 'Instrument_type': 'Banjo', >>> 'Mission_group': 'Impossible', >>> 'Logical_source': ':)', >>> 'Logical_source_description': ':('}
>>> # Lets add a new coordinate variable for the "direction" >>> dir_data = [1, 2, 3] >>> dir_dims = ['direction'] >>> direction = xr.Variable(dir_dims, dir_data)
>>> # Recreate the Dataset with this new objects, and recreate the CDF >>> ds = xr.Dataset(data_vars={'data': data, 'epoch': epoch, 'direction':direction}, attrs=global_attributes) >>> os.remove('hello.cdf') >>> cdflib.xarray_to_cdf(ds, 'hello.cdf')
- Example netCDF -> CDF conversion:
>>> # Download a netCDF file (if needed) >>> fname = 'dn_magn-l2-hires_g17_d20211219_v1-0-1.nc' >>> url = ("https://lasp.colorado.edu/maven/sdc/public/data/sdc/web/cdflib_testing/dn_magn-l2-hires_g17_d20211219_v1-0-1.nc") >>> if not os.path.exists(fname): >>> urllib.request.urlretrieve(url, fname)
>>> # Load in the dataset, and set VAR_TYPES attributes (the most important attribute as far as this code is concerned) >>> goes_r_mag = xr.load_dataset("C:/Work/cdf_test_files/dn_magn-l2-hires_g17_d20211219_v1-0-1.nc") >>> for var in goes_r_mag: >>> goes_r_mag[var].attrs['VAR_TYPE'] = 'data' >>> goes_r_mag['coordinate'].attrs['VAR_TYPE'] = 'support_data' >>> goes_r_mag['time'].attrs['VAR_TYPE'] = 'support_data' >>> goes_r_mag['time_orbit'].attrs['VAR_TYPE'] = 'support_data'
>>> # Create the CDF file >>> cdflib.xarray_to_cdf(goes_r_mag, 'hello.cdf')
- Processing Steps:
- Determines the list of dimensions that represent time-varying dimensions. These ultimately become the “records” of the CDF file
If it is named “epoch” or “epoch_N”, it is considered time-varying
If a variable points to another variable with a DEPEND_0 attribute, it is considered time-varying
If a variable has an attribute of VAR_TYPE equal to “data”, it is time-varying
If a variable has an attribute of VAR_TYPE equal to “support_data” and it is 2 dimensional, it is time-varying
- Determine a list of “dimension” variables within the Dataset object
These are all coordinates in the dataset that are not time-varying
Additionally, variables that a DEPEND_N attribute points to are also considered dimensions
Optionally, if ISTP=true, automatically add in DEPEND_0/1/2/etc attributes as necessary
Optionally, if ISTP=true, check all variable attributes and global attributes are present
Convert all data into either CDF_INT8, CDF_DOUBLE, CDF_UINT4, or CDF_CHAR
Optionally, convert variables with the name “epoch” or “epoch_N” to CDF_TT2000
Write all variables and global attributes to the CDF file!
- ISTP Warnings:
If ISTP=true, these are some of the common things it will check:
Missing or invalid VAR_TYPE variable attributes
DEPEND_N missing from variables
DEPEND_N/LABL_PTR/UNIT_PTR/FORM_PTR are pointing to missing variables
Missing required global attributes
Missing an “epoch” dimension
DEPEND_N attribute pointing to a variable with oncompatible dimensions