ctdam.parser package

Submodules

• ctdam.parser.casts module
  • Casts
    • Casts.data
    • Casts.processing_info
    • Casts.path_to_data
    • Casts.anomalous_data
    • Casts.cruise
    • Casts.sensor_info
    • Casts.plot_dir
    • Casts.convert()
    • Casts.check_converted_data()
    • Casts.read_sensor_info()
    • Casts.process()
    • Casts.plot()
    • Casts.to_tsv()
• ctdam.parser.custom_xarray_accessors module
  • ProcessingAccessor
    • ProcessingAccessor.module()
    • ProcessingAccessor.workflow()
    • ProcessingAccessor.last
  • InputAccessor
    • InputAccessor.parameter()
    • InputAccessor.bottles()
    • InputAccessor.processing_metadata()
    • InputAccessor.teos10_vars()
  • MetadataAccessor
    • MetadataAccessor.provenance
    • MetadataAccessor.custom
  • DataRetrievalAccessor
    • DataRetrievalAccessor.btl_info
    • DataRetrievalAccessor.spans()
    • DataRetrievalAccessor.size
    • DataRetrievalAccessor.sample_rate
    • DataRetrievalAccessor.bin_unit
    • DataRetrievalAccessor.binned
    • DataRetrievalAccessor.sensor_strand()
    • DataRetrievalAccessor.flattened_ds()
    • DataRetrievalAccessor.numpy_array()
    • DataRetrievalAccessor.pandas_dataframe
  • ExportAccessor
    • ExportAccessor.to_cnv()
    • ExportAccessor.to_btl()
  • QCAccessor
    • QCAccessor.set_flag()
    • QCAccessor.edit_value()
    • QCAccessor.masked()
    • QCAccessor.check_sensor_agreement()
    • QCAccessor.best_estimate()
  • PlotAccessor
    • PlotAccessor.profile()
    • PlotAccessor.flagged()
• ctdam.parser.geomar_ctd_file_parser module
  • GEOMARCTDFile
    • GEOMARCTDFile.read_file()
    • GEOMARCTDFile.create_dataframe()
• ctdam.parser.read_ctd_data module
  • create_array_coords()
  • create_array_attrs()
  • sorting_parameters()
  • parse_oxygen_data()
  • read_cnv()
  • build_sensor_pairs()
  • read_hex()
  • read_ctd_data()
• ctdam.parser.seabird_data_files module
  • SeabirdDataFile
    • SeabirdDataFile.path_to_file
    • SeabirdDataFile.file_name
    • SeabirdDataFile.file_dir
    • SeabirdDataFile.data
    • SeabirdDataFile.metadata
    • SeabirdDataFile.start_time
    • SeabirdDataFile.start_position
    • SeabirdDataFile.cruise
    • SeabirdDataFile.station
    • SeabirdDataFile.event_name
    • SeabirdDataFile.read_file()
    • SeabirdDataFile.reading_start_time()
    • SeabirdDataFile.reading_start_position()
    • SeabirdDataFile.read_event_information()
    • SeabirdDataFile.sensor_xml_to_flattened_dict()
    • SeabirdDataFile.structure_metadata()
  • CnvFile
    • CnvFile.parse_cnv_data_format()
    • CnvFile.absolute_time_calculation()
  • HexFile
    • HexFile.parse_hex()
    • HexFile.get_corresponding_xmlcon()
  • BottleLogFile
    • BottleLogFile.data_whitespace_removal()
    • BottleLogFile.obtaining_reset_time()
    • BottleLogFile.create_dataframe()
  • BottleFile
    • BottleFile.create_dataframe()
    • BottleFile.adding_timestamp_column()
    • BottleFile.selecting_rows()
    • BottleFile.reading_data_header()
• ctdam.parser.sst_ctd_file_parser module
  • sst2xarray()
• ctdam.parser.xmlfiles module
  • XMLFile
    • XMLFile.to_xml()
    • XMLFile.to_json()
  • XMLCONFile
    • XMLCONFile.read_xml_config()
    • XMLCONFile.xml_coeffs_to_float()
    • XMLCONFile.get_sensor_info()
  • PsaFile

Module contents

class ctdam.parser.Casts(path_to_data='', ctd_data=[], processing_info={}, pattern='', plot=False, show_plot=True, plot_dir='htmls')

Bases: UserList

A structure to ease working with multiple ctd casts.

Can work with ascii files, converted (.cnv) or non-converted (.hex), as well as data within python objects (CTDData). Automates the very basic actions ussually performs on these data: converting, processing, plotting and exporting. The cpu-heavy actions (convertion and processing) are calculated in parallel, using multithreading.

Parameters:

• path_to_data (Path | str) – Path to target files

• ctd_data (list[Dataset]) – A list of target CTDData objects

• processing_info (dict) – Processing configuration

• pattern (str) – A file pattern to filter the target files with

• plot (bool) – Whether to create .html plots of the target files

• show_plot (bool) – Whether to display the plots in a browser

• plot_dir (Path | str) – The directory to store the plots in

data

A list of CTDData objects

Type:

list

processing_info

A processing workflow as used in Procedure

Type:

dict

path_to_data

The input path to the data file(s)

Type:

Path

anomalous_data

A list of data files that seem to be suspiciously small

Type:

list

cruise

The name of the cruise the data files were collected in

Type:

str

sensor_info

The sensor structure as generated by get_unique_sensor_data()

Type:

list

plot_dir

A file path where generated data plots are saved to

Type:

Path

convert(file)

Converts .hex files.

Can work with hex2py processing settings and hides all warnings.

Parameters:

file (Path) – Path to target hex files

check_converted_data()

Basic output data size check.

Catches very obvious test or wrong CTD data. These are saved in an anomalies attribute to allow human intervention.

Return type:

list[Dataset]

read_sensor_info()

Parses all sensor metadata in one structure.

Usually, the sensor layout does not change during one cruise. But if it does, this function should detect that change and document it with a new list that displays the cast number and the differing sensor metadata.

process(processing_info, target_files=[])

Applies the given processing workflow to all CTD data.

Uses multiprocessing for parallel processing and tqdm to display the progress.

Parameters:

• processing_info (dict) – Processing workflow configuration

• target_files (list[Dataset]) – The files to process

plot(show_plot=True)

Creates .html plots of the target data.

Uses bokeh for interactive plotting and creates one ‘main’ .html that incorporates all individual cast plots.

Parameters:

show_plot (bool) – Whether to open the plot in a browser

to_tsv(file_name=None)

Exports the target file data into one great .tsv file.

Parameters:

file_name (str | Path | None) – The name of the exported file

ctdam.parser.read_ctd_data(path_to_ctd_data_file)

Parse different file types to a cf-compliant xarray Dataset

Parameters:

path_to_ctd_data_file (Path | str) – The path to the ctd data file

Return type:

Dataset

ctdam.parser.read_cnv(path_to_cnv_file, only_header=False)

Parse Seabird .cnv data to cf-compliant xarray Dataset.

Parameters:

• path_to_cnv_file (Path | str) – The path to the .cnv file

• only_header (bool) – Whether to only parse the header information

Return type:

Dataset

ctdam.parser.read_hex(path_to_hex_file)

Parse Seabird .hex data to cf-compliant xarray Dataset.

Parameters:

path_to_cnv_file (Path | str :) – The path to the .hex file

Return type:

Dataset

class ctdam.parser.CnvFile(path_to_file, only_header=False)

Bases: SeabirdDataFile

A representation of a cnv-file as used by SeaBird.

parse_cnv_data_format()

Return type:

dict[str, ndarray]

absolute_time_calculation()

Replaces the basic cnv time representation of counting relative to the casts start point, by a unix timestamp.

Return type:

ndarray

class ctdam.parser.HexFile(path_to_file, path_to_xmlcon='', *args, **kwargs)

Bases: SeabirdDataFile

A representation of a .hex file as used by SeaBird.

When no corresponding .xmlcon file given, a search algorithm is used to determine the matching .xmlcon automatically.

parse_hex(hex)

Parse the individual hex information bits using sbe.odf

Parameters:

hex (Path | str) – The path to the target hex file

Return type:

Dataset

get_corresponding_xmlcon(path_to_xmlcon='')

Finds the best matching .xmlcon file inside the same directory.

The logics works as follows:

• if an .xmlcon of the same name exists, take that

• else, find all .xmlcons of the same cruise inside the given directory and use the one used by the previous .hex file, sorted by file name.

Return type:

XMLCONFile | None

class ctdam.parser.BottleFile(path_to_file)

Bases: SeabirdDataFile

Class that represents a Sea-Bird Bottle File (.btl) .

create_dataframe()

Creates a dataframe out of the .btl file. Handles the double data header correctly.

adding_timestamp_column()

Creates a timestamp column that holds both, Date and Time information.

selecting_rows(df=None, statistic_of_interest=['avg'])

Creates a dataframe with the given row identifier, using the statistics column. A single string or a list of strings can be processed.

Parameters:

• df (pandas.Dataframe :) – the files Pandas representation (Default value = self.df)

• statistic_of_interest (list | str) – collection of values of the ‘statistics’ column in self.df

reading_data_header()

Identifies and separatly collects the rows that specify the data tables headers.

class ctdam.parser.BottleLogFile(path_to_file)

Bases: SeabirdDataFile

Bottle Log file (.bl) representation, that extracts the three different data types from the file: reset time and the table with bottle IDs and corresponding data ranges.

data_whitespace_removal()

Strips the input from whitespace characters, in this case especially newline characters.

Return type:

list

obtaining_reset_time()

Reading reset time with small input check.

Return type:

datetime

create_dataframe()

Creates a dataframe from the list specified in self.data.

Return type:

DataFrame

class ctdam.parser.XMLCONFile(path_to_file)

Bases: XMLFile

A representation of a Sea-Bird .XMLCON file.

read_xml_config()

Parse the companion .xmlcon calibration file into self.cfgp.

Locates the xmlcon file alongside the hex file, parses the SensorArray block, and converts coefficient strings to floats. Sensors not in the supported set are skipped.

xml_coeffs_to_float(cfgp)

Returns float-parsed xml coefficients.

get_sensor_info()

Creates a multilevel dictionary, dropping the first four dictionaries, to retrieve pure sensor information.

Return type:

list[dict]