ctdam package

Subpackages

• ctdam.conv package
  • Submodules
    • ctdam.conv.cast_borders module
      • smoothing()
      • get_cast_borders()
      • get_downcast_end()
      • get_downcast_start()
      • get_upcast_start()
      • get_upcast_end()
      • soaking_detection()
      • soaking_removal()
      • combine_downcast_start()
    • ctdam.conv.raw_conversion module
      • calculate_tau()
      • calculate_dvdt_window()
      • calculate_oxsol()
      • pressure()
      • temperature()
      • conductivity()
      • salinity()
      • convert_sbe43_oxygen()
      • oxygen()
      • par_biosphericallicorchelsea()
      • altimeter()
      • fluorescence()
      • turbidity()
      • fluorowetlabcdom()
      • spar()
    • ctdam.conv.unit_conversion module
      • oxygen_mlperl_to_umolperkg()
      • oxygen_mlperl_to_umolperl()
      • oxygen_umolperkg_to_umolperl()
      • oxygen_umolperl_to_umolperkg()
      • get_potential_density()
  • Module contents
    • get_cast_borders()
    • oxygen_umolperl_to_umolperkg()
    • oxygen_mlperl_to_umolperkg()
    • oxygen_umolperkg_to_umolperl()
    • oxygen_mlperl_to_umolperl()
    • get_potential_density()
• ctdam.parser package
  • Submodules
    • ctdam.parser.casts module
      • Casts
    • ctdam.parser.custom_xarray_accessors module
      • ProcessingAccessor
      • InputAccessor
      • MetadataAccessor
      • DataRetrievalAccessor
      • ExportAccessor
      • QCAccessor
      • PlotAccessor
    • ctdam.parser.geomar_ctd_file_parser module
      • GEOMARCTDFile
    • ctdam.parser.read_ctd_data module
      • create_array_coords()
      • create_array_attrs()
      • sorting_parameters()
      • parse_oxygen_data()
      • read_cnv()
      • build_sensor_pairs()
      • read_hex()
      • read_ctd_data()
    • ctdam.parser.seabird_data_files module
      • SeabirdDataFile
      • CnvFile
      • HexFile
      • BottleLogFile
      • BottleFile
    • ctdam.parser.sst_ctd_file_parser module
      • sst2xarray()
    • ctdam.parser.xmlfiles module
      • XMLFile
      • XMLCONFile
      • PsaFile
  • Module contents
    • Casts
      • Casts.data
      • Casts.processing_info
      • Casts.path_to_data
      • Casts.anomalous_data
      • Casts.cruise
      • Casts.sensor_info
      • Casts.plot_dir
      • Casts.convert()
      • Casts.check_converted_data()
      • Casts.read_sensor_info()
      • Casts.process()
      • Casts.plot()
      • Casts.to_tsv()
    • read_ctd_data()
    • read_cnv()
    • read_hex()
    • CnvFile
      • CnvFile.parse_cnv_data_format()
      • CnvFile.absolute_time_calculation()
    • HexFile
      • HexFile.parse_hex()
      • HexFile.get_corresponding_xmlcon()
    • BottleFile
      • BottleFile.create_dataframe()
      • BottleFile.adding_timestamp_column()
      • BottleFile.selecting_rows()
      • BottleFile.reading_data_header()
    • BottleLogFile
      • BottleLogFile.data_whitespace_removal()
      • BottleLogFile.obtaining_reset_time()
      • BottleLogFile.create_dataframe()
    • XMLCONFile
      • XMLCONFile.read_xml_config()
      • XMLCONFile.xml_coeffs_to_float()
      • XMLCONFile.get_sensor_info()
• ctdam.proc package
  • Subpackages
    • ctdam.proc.modules package
      • Submodules
      • Module contents
  • Submodules
    • ctdam.proc.entry module
      • process()
    • ctdam.proc.module module
      • Module
    • ctdam.proc.settings module
      • Configuration
      • IncompleteProcedureConfig
    • ctdam.proc.utils module
      • default_seabird_exe_path()
      • is_directly_measured_value()
      • fill_file_type_dir()
    • ctdam.proc.workflow module
      • Workflow
  • Module contents
    • process()
    • is_directly_measured_value()
    • fill_file_type_dir()
    • Workflow
      • Workflow.load_config()
      • Workflow.check_config_entry()
      • Workflow.new_file_path()
      • Workflow.run()
• ctdam.qc package
  • Submodules
    • ctdam.qc.quality_flags module
      • SeaDataNetFlag
    • ctdam.qc.range_checks module
      • RangeLimit
      • apply_range_check_to_parameter()
      • apply_range_check()
      • apply_default_range_checks()
    • ctdam.qc.spike_checks module
      • SpikeLimit
      • apply_spike_check_to_parameter()
      • apply_spike_check()
      • apply_default_spike_checks()
  • Module contents
• ctdam.vis package
  • Submodules
    • ctdam.vis.visualize module
      • check_and_create_path()
      • cruise_plots()
      • basic_bokeh_plot()
      • create_main_html()
  • Module contents
    • cruise_plots()
    • create_main_html()
    • basic_bokeh_plot()

Submodules

• ctdam.exceptions module
  • BinnedDataError
  • UnexpectedFileFormat
  • MissingParameterError
  • InvalidArgumentCombination
  • NoDataError
• ctdam.utils module
  • read_event_name()
  • create_event_string()
  • parse_xmlcon_sensor_data()
  • extract_sensor_name()
  • get_unique_sensor_data()
  • coordinates_to_float()
  • sbe_to_decimal()
  • map_metadata()

Module contents

ctdam.get_cast_borders(pressure, downcast_only=True, min_size_factor=0.01, min_soak_window=100, max_fd_quotient=6, prominence_divisor=7, win_size_divisor=500, min_velocity_quotient=15, min_velocity=0.045)

Calculates start and end points of one CTD cast.

Uses first (fd) and second derivatives (sd) for that. Relies on carefully fine-tuned parameters that are set as default values. These can be fit to any kind of CTD data.

Parameters:

• pressure (ndarray) – Pressure array

• downcast_only (bool) – Whether to only work with downcast data (Default value = True)

• min_size_factor (float) – Factor to check final dataset size against (Default value = 0.01)

• min_soak_window (int) – Downcast_start: minimum size of soaking window (Default value = 100)

• max_fd_quotient (int) – Downcast_start: Cut-off of fd height (Default value = 6)

• prominence_divisor (int) – Downcast_start: Minimum size of sd peak prominence (Default value = 7)

• win_size_divisor (int) – Downcast_start: Search window size to check fd means (Default value = 500)

• min_velocity_quotient (int) – Downcast_start: Minimum velocity cut-off (Default value = 15)

• min_velocity (float) – Downcast_start: Minimum velocity cut-off (Default value = 0.045)

Return type:

dict

ctdam.oxygen_mlperl_to_umolperkg(data, potential_density)

Return type:

ndarray

ctdam.oxygen_mlperl_to_umolperl(data)

Return type:

ndarray

ctdam.oxygen_umolperkg_to_umolperl(data, potential_density)

Return type:

ndarray

ctdam.oxygen_umolperl_to_umolperkg(data, potential_density)

Return type:

ndarray

ctdam.get_potential_density(practical_salinity, temperature, pressure, longitude, latitude)

Return type:

ndarray

class ctdam.Casts(path_to_data='', ctd_data=[], processing_info={}, pattern='', plot=False, show_plot=True, plot_dir='htmls')

Bases: UserList

A structure to ease working with multiple ctd casts.

Can work with ascii files, converted (.cnv) or non-converted (.hex), as well as data within python objects (CTDData). Automates the very basic actions ussually performs on these data: converting, processing, plotting and exporting. The cpu-heavy actions (convertion and processing) are calculated in parallel, using multithreading.

Parameters:

• path_to_data (Path | str) – Path to target files

• ctd_data (list[Dataset]) – A list of target CTDData objects

• processing_info (dict) – Processing configuration

• pattern (str) – A file pattern to filter the target files with

• plot (bool) – Whether to create .html plots of the target files

• show_plot (bool) – Whether to display the plots in a browser

• plot_dir (Path | str) – The directory to store the plots in

data

A list of CTDData objects

Type:

list

processing_info

A processing workflow as used in Procedure

Type:

dict

path_to_data

The input path to the data file(s)

Type:

Path

anomalous_data

A list of data files that seem to be suspiciously small

Type:

list

cruise

The name of the cruise the data files were collected in

Type:

str

sensor_info

The sensor structure as generated by get_unique_sensor_data()

Type:

list

plot_dir

A file path where generated data plots are saved to

Type:

Path

convert(file)

Converts .hex files.

Can work with hex2py processing settings and hides all warnings.

Parameters:

file (Path) – Path to target hex files

check_converted_data()

Basic output data size check.

Catches very obvious test or wrong CTD data. These are saved in an anomalies attribute to allow human intervention.

Return type:

list[Dataset]

read_sensor_info()

Parses all sensor metadata in one structure.

Usually, the sensor layout does not change during one cruise. But if it does, this function should detect that change and document it with a new list that displays the cast number and the differing sensor metadata.

process(processing_info, target_files=[])

Applies the given processing workflow to all CTD data.

Uses multiprocessing for parallel processing and tqdm to display the progress.

Parameters:

• processing_info (dict) – Processing workflow configuration

• target_files (list[Dataset]) – The files to process

plot(show_plot=True)

Creates .html plots of the target data.

Uses bokeh for interactive plotting and creates one ‘main’ .html that incorporates all individual cast plots.

Parameters:

show_plot (bool) – Whether to open the plot in a browser

to_tsv(file_name=None)

Exports the target file data into one great .tsv file.

Parameters:

file_name (str | Path | None) – The name of the exported file

ctdam.read_ctd_data(path_to_ctd_data_file)

Parse different file types to a cf-compliant xarray Dataset

Parameters:

path_to_ctd_data_file (Path | str) – The path to the ctd data file

Return type:

Dataset

ctdam.read_cnv(path_to_cnv_file, only_header=False)

Parse Seabird .cnv data to cf-compliant xarray Dataset.

Parameters:

• path_to_cnv_file (Path | str) – The path to the .cnv file

• only_header (bool) – Whether to only parse the header information

Return type:

Dataset

ctdam.read_hex(path_to_hex_file)

Parse Seabird .hex data to cf-compliant xarray Dataset.

Parameters:

path_to_cnv_file (Path | str :) – The path to the .hex file

Return type:

Dataset

class ctdam.CnvFile(path_to_file, only_header=False)

Bases: SeabirdDataFile

A representation of a cnv-file as used by SeaBird.

parse_cnv_data_format()

Return type:

dict[str, ndarray]

absolute_time_calculation()

Replaces the basic cnv time representation of counting relative to the casts start point, by a unix timestamp.

Return type:

ndarray

class ctdam.HexFile(path_to_file, path_to_xmlcon='', *args, **kwargs)

Bases: SeabirdDataFile

A representation of a .hex file as used by SeaBird.

When no corresponding .xmlcon file given, a search algorithm is used to determine the matching .xmlcon automatically.

parse_hex(hex)

Parse the individual hex information bits using sbe.odf

Parameters:

hex (Path | str) – The path to the target hex file

Return type:

Dataset

get_corresponding_xmlcon(path_to_xmlcon='')

Finds the best matching .xmlcon file inside the same directory.

The logics works as follows:

• if an .xmlcon of the same name exists, take that

• else, find all .xmlcons of the same cruise inside the given directory and use the one used by the previous .hex file, sorted by file name.

Return type:

XMLCONFile | None

class ctdam.BottleFile(path_to_file)

Bases: SeabirdDataFile

Class that represents a Sea-Bird Bottle File (.btl) .

create_dataframe()

Creates a dataframe out of the .btl file. Handles the double data header correctly.

adding_timestamp_column()

Creates a timestamp column that holds both, Date and Time information.

selecting_rows(df=None, statistic_of_interest=['avg'])

Creates a dataframe with the given row identifier, using the statistics column. A single string or a list of strings can be processed.

Parameters:

• df (pandas.Dataframe :) – the files Pandas representation (Default value = self.df)

• statistic_of_interest (list | str) – collection of values of the ‘statistics’ column in self.df

reading_data_header()

Identifies and separatly collects the rows that specify the data tables headers.

class ctdam.BottleLogFile(path_to_file)

Bases: SeabirdDataFile

Bottle Log file (.bl) representation, that extracts the three different data types from the file: reset time and the table with bottle IDs and corresponding data ranges.

data_whitespace_removal()

Strips the input from whitespace characters, in this case especially newline characters.

Return type:

list

obtaining_reset_time()

Reading reset time with small input check.

Return type:

datetime

create_dataframe()

Creates a dataframe from the list specified in self.data.

Return type:

DataFrame

class ctdam.XMLCONFile(path_to_file)

Bases: XMLFile

A representation of a Sea-Bird .XMLCON file.

read_xml_config()

Parse the companion .xmlcon calibration file into self.cfgp.

Locates the xmlcon file alongside the hex file, parses the SensorArray block, and converts coefficient strings to floats. Sensors not in the supported set are skipped.

xml_coeffs_to_float(cfgp)

Returns float-parsed xml coefficients.

get_sensor_info()

Creates a multilevel dictionary, dropping the first four dictionaries, to retrieve pure sensor information.

Return type:

list[dict]

ctdam.process(input='', modules=['loop_removal', 'wildedit_geomar', 'wfilter', 'alignctd', 'celltm', 'binavg'], other_settings={}, use_multiprocessing=True, **kwargs)

Return type:

Dataset | List[Dataset]

ctdam.is_directly_measured_value(parameter)

Returns whether a parameter has been measured via a sensor or is calculated.

Return type:

bool

ctdam.fill_file_type_dir(file_type_dir, file, copy=True)

Copies the target input and output files into individual type directories.

A ‘file type directory’ is a directory that is meant to collect all the file of the same file extension that accumulate over multiple processings. For typical Sea-Bird processings you usually end up with something like this:

root-dir

• hex

• cnv

• XMLCON

• btl

• bl

• hdr

Parameters:

• file (Path)

• copy (bool) – (Default value = True)

class ctdam.Workflow(ds, configuration, auto_run=True)

Bases: object

Runs a couple of processing steps in sequence on one or more CTD data source files.

It can use seabird internal processing modules, as well as custom ones. These can be in the form of independent windows exes or just pure python code. The input data can be .hex, .cnv or python data representations, predominantly CTDData. The input and all module and extra information is stored in a dict that usually will be generated by the settings Configuration module that reads a toml config.

Parameters:

• configuration (dict | Configuration) – The information necessary to run a processing procedure.

• auto_run (bool) – Whether to autopilot the whole procedure.

load_config()

Thorough input/format check of the processing configuration, that either stems from a .toml config file, or is a self-build dictionary. Checks for the presence of certain keys, and then, depending on their importance, either fails or sets default values.

check_config_entry(key, default_value)

Handles configuration file entries.

new_file_path(file=PosixPath('.'))

Creates the new output file path.

Takes the file type directory or the given output directory and joins them with the given output name.

Parameters:

file (Path) – The current path to the target file.

Return type:

Path

run()

Performs the processing on all target files.

This is the ‘main’ method of the procedure. All previous methods prepare data for this method to then finally transform the input files into the wanted format. The main purpose of this method is the coordination of the two different forms of processing modules: standalone executables with config files, mainly Sea-Bird processing modules, and python-internal classes that implement the Module interface. The caveats are mainly the switching from one form to the other. This for example results in a in or out parsing of a CnvFile object.

Return type:

Dataset

ctdam.cruise_plots(directory='', output_directory='html', output_name='main.html', embed_contents=False, html_title='', overwrite=False, no_new_plots=False, size_limit=10, filter='', show_html=True, config_path='vis_config.toml', file_type='cnv')

Run basic_bokeh_plot and create_main_html and handle inputs.

Parameters:

• directory (Path | str) – The directory to look for data files to plot (Default value = “”)

• output_directory (Path | str) – The directory to save .html file to (Default value = “html”)

• output_name (str) – The name of the main html file (Default value = “main.html”)

• embed_contents (bool) – Whether to embed plot htmls into main html (Default value = False)

• html_title (str) – The header of the main html (Default value = “”)

• overwrite (bool) – Whether to overwrite an existing main html (Default value = False)

• no_new_plots (bool) – Whether to not overwrite existing plot htmls (Default value = False)

• size_limit (int) – Data file size limit in MB (Default value = 10)

• filter (str) – A search filter for files (Default value = “”)

• show_html (bool) – Whether to open main html in browser (Default value = True)

• config_path (Path | str) – The path to vis configuration info (Default value = “vis_config.toml”)

• file_type (str) – The file type to search for (Default value = “cnv”)

Return type:

Path | None

ctdam.create_main_html(directory_path, output_name='main_plots.html', output_directory='', embed_contents=True, title='', show_html=True)

Assemble a main .html file that stores all individual .html plots.

Does also allow to interactively change plotting parameters and the seamless selection of plots.

Parameters:

• directory_path (Path | str) – The path to the .html plot files

• output_name (str) – The name of the main .html file (Default value = “main_plots.html”)

• output_directory (Path | str) – The directory to write the main .html file to (Default value = “”)

• embed_contents (bool) – Whether to embed the .html plots into the main html file (Default value = True)

• title (str) – The title of the main file (Default value = “”)

• show_html (bool) – Whether to open the main .html in a browser (Default value = True)

Return type:

Path | None

ctdam.basic_bokeh_plot(ctd_data, print_plot=False, output_name='', output_directory='', metadata=True, show_plot=True, y_axis_params=['pressure', 'depth'], config_path='vis_config.toml')

Create a .html plot for a CTD cast.

Parameters:

• ctd_data (Path | str | Dataset) – The data to operate on

• print_plot (bool) – Whether to save the plot to disk (Default value = False)

• output_name (str) – The name of the output file (Default value = “”)

• output_directory (Path | str) – The directory to store the output file in (Default value = “”)

• metadata (bool) – Whether to save metadata in the file (Default value = True)

• show_plot (bool) – Whether to open the plot in a browser (Default value = True)

• y_axis_params (list[str]) – Possible parameters for the y axis

• config_path (Path | str) – The path to the config file (Default value = “vis_config.toml”)