Usage

Installation

This software can be used in a few different ways. To run processing workflows, called ‘procedures’, there is a command line cli, that resides in main.py and runs workflow files on given CTD data files.

The easiest way to install CLI interface called ‘ctdam’ is to use uv or pipx:

$ uv tool install --from 'ctdam[all]' ctdam

$ uv run ctdam check
All set, you are ready to go.

$ pipx install ctdam

$ ctdam check
All set, you are ready to go.

Otherwise, via basic python package managers, like pip, conda or poetry:

# using pip
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install ctd-processing
$ python src/processing/main.py check
All set, you are ready to go.

# using poetry
$ poetry install
$ poetry env activate
$ ctdam check
All set, you are ready to go.

CLI

The CLI ‘ctdam’ should be the main entry point to the software. It allows running and editing of procedure workflows. It features the following commands:

$ ctdam --help

Usage: ctdam [OPTIONS] COMMAND [ARGS]...

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ --verbose             -v        Enable verbose output.                                                 │
│ --install-completion            Install completion for the current shell.                              │
│ --show-completion               Show completion for the current shell, to copy it or customize the     │
│                                 installation.                                                          │
│ --help                          Show this message and exit.                                            │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────╮
│ run      Processes one target file using the given procedure workflow file.                            │
│ convert  Converts a list of Sea-Bird raw data files (.hex) to .cnv files.                              │
│          Does either use an explicit list of paths or searches for all .hex files in                   │
│          the given directory.                                                                          │
│ batch    Applies a processing config to multiple .hex or. cnv files.                                   │
│ edit     Opens a procedure workflow file in GUI for editing.                                           │
│ show     Display the contents of a procedure workflow file.                                            │
│ plot     Plot a cnv file.                                                                              │
│ vis      Create a main html that incorporates the individual .html plots.                              │
│ check    Assures that all requirements to use this tool are met.                                       │
│ log      Prints the last x entries of the log file.                                                    │
│ version  Displays the version number of this software.                                                 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯

The top three commands are the commands to run processing workflows, while the other ones are for editing workflow files and plotting. In general, their description should be self-explanatory.

To get more specific information to the individual commands and their arguments, you can display help sites for each of them:

$ ctdam batch --help

Usage: ctdam batch [OPTIONS] INPUT_DIR CONFIG

Applies a processing config to multiple .hex or. cnv files.

╭─ Arguments ────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    input_dir      TEXT  The data directory with the target files. [required]                         │
│ *    config         TEXT  Either an explicit config as dict or a path to a .toml config file.          │
│                           [required]                                                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ --pattern  -p      TEXT  A name pattern to filter the target files with. [default: .hex]               │
│ --help                   Show this message and exit.                                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────

Processing workflows

The workflow files configure everything needed to run a processing procedure and are easy to understand and write.

Another way is to run processing procedures directly from your python code, using either workflow files or the fact that these workflows correspond to a basic python dictionary internally. This allows for quick scripts that can use flexible input variables.

Warning

To run Sea-Bird processing modules, you need a local installation of them, from their website.

from ctdam.proc import Procedure

# using own dictionary:
 proc_config = {
     "psa_directory": "path_to_psas",
     "output_dir": "path_to_output",
     "modules": {
         "datcnv": {"psa": "DatCnv.psa"},
         "wildedit_geomar": {"std1": 3.0, "window_size": 100},
         "celltm": {"psa": ("CellTM.psa")},
     },
 }
procedure = Procedure(proc_config)

# using a .toml workflow file:
procedure = Procedure("example_config.toml")

The full list of options for running processing procedures, can be obtained from its docstring:

class ctdam.proc.procedure.Procedure(configuration, seabird_exe_directory=None, available_hex_converters=['datcnv', 'hex2py'], auto_run=True, procedure_fingerprint_directory=None, file_type_dir=None, plot=False, verbose=False, timeout=60)[source]

Runs a couple of processing steps in sequence on one or more CTD data source files.

It can use seabird internal processing modules, as well as custom ones. These can be in the form of independent windows exes or just pure python code. The input data can be .hex, .cnv or python data representations, predominantly CTDData. The input and all module and extra information is stored in a dict that usually will be generated by the settings Configuration module that reads a toml config.

Parameters:

  • configuration (dict | Configuration) – The information necessary to run a processing procedure.

  • seabird_exe_directory (Path | str | None) – The path to the directory where the Sea-Bird exes reside in.

  • available_hex_converters (list[str]) – A list of the known hex converters.

  • auto_run (bool) – Whether to autopilot the whole procedure.

  • procedure_fingerprint_directory (Path | str | None) – A path to a directory where the fingerprint are meant to be stored in. If none given, this option is considered to be turned off.

  • file_type_dir (Path | str | None) – A path to a directory where the individual Sea-Bird file types are differentiated into respective directories. If none given, this option is considered to be turned off.

  • plot (bool) – Whether to create .html plots of the output data

  • verbose (bool) – Sets whether the Sea-Bird modules are run silently or not.

  • timeout (int) – The time in seconds after which individual processing steps will be killed automatically.

Workflow files

The workflow dictionary used in the example above corresponds to this .toml workflow file:

# example_config.toml

psa_directory = "path_to_psas"
output_dir = "path_to_output"

[modules.datcnv]
psa = "DatCnv.psa"

[modules.wildedit_geomar]
std1 = 3.0
window_size = 100

[modules.celltm]
psa = "CellTM.psa"

It does not matter, which way you provide the processing workflow information, the two above examples work in exactly the same way. The different options available and how to use them, are shown below:

proc_template.toml
# path to a target file
input = ""
# the directory where all psas for this workflow reside
psa_directory = ""
# path to target xmlcon if not in the same directory with the same name as hex
xmlcon = ''
# the value to flag bad values with
bad_flag = -9.990e-29
# whether to cut the downcast and work only on that
downcast_only = true
# whether to remove data rows with bad flag
remove_flags = true
# whether to write to disk
output_type = 'cnv'
# the name of the new file, if none, the input files name will be used
output_name = ''
# the path to the output directory to store the new file in, if none, the input
# directory will be used
output_dir = '.'
# the parameters to print to the file, non-existent ones will be skipped
output_columns = [
    "Pressure",
    "Temperature",
    "Salinity",
    "Oxygen",
    "Fluorescence",
    "Turbidity",
    "PAR",
    "SPAR",
    'Time',
    "Latitude",
    "Longitude",
]
# the individual modules to run on the input data
## to run Sea-Bird processing executables, the key 'psa' is needed for
## distinguising.
[modules.wildedit_geomar]

[modules.wfilter]

[modules.alignctd]

[modules.celltm]

[modules.create_bottle_file]

# all gsw function names are understood and can be used here
[modules.entropy_from_CT]

[modules.binavg]

Tip

The only mandatory option is the modules dictionary, describing the steps to run on the file(s). All other options are optional.

A simple gui can be used to write and edit workflow files, which, at the moment, is run like this:

ctdam edit example_config.toml
_images/procedure_gui.png

Note

All options for the Procedure class can be found here.