Core and support

The top-level package and the shared support modules used by every table kind.

metadata_tools

PDS Ring-Moon Systems Node, SETI Institute

metadata-tools is a Python module that generates index and geometry metadata tables and their corresponding PDS3 labels. Each line of the table contains metadata for a single data file (e.g. image).

Index files contain descriptive information about the data product, like observation times, exposures, instrument modes and settings, etc. Index file entries are taken from the label for the data product by default, but may instead be derived from label quantities by defining the appropriate configuration function in the host_config.py for the specific host.

Raw index files are provided by each project, with varying levels of compliance. The project-supplied index files are modified to produce the corrected index files that can be used with the host from_index() method. The metadata-tools package is intended to produce supplemental index files, which add columns to the corrected index file. Supplemental index files are identical in structure to index files, so this package can generate any kind of index file. Supplemental index files can be provided as arguments to from_index() to create a merged dictionary.

Supplemental index files are used as input to OPUS, and are available via viewmaster to be downloaded by PDS users.

Geometry files tabulate the values of geometric quantities for each data file, which are derived from SPICE using the information in the index file or from the PDS3 label using OOPS. The purpose of the geometry files is to provide input to OPUS.

Running the code

The procedure for generating metadata tables for a given collection is as follows:

1. Create the supplemental index using <collection>_index.py.

2. Create the geometry tables using <collection>_geometry.py.

3. Generate the cumulative tables using <collection>_cumulative.py.

Generating New Metadata Tables

The procedure for creating a new host configuration is as follows:

1. Create a directory for the new host collection under the hosts/ subdirectory, e.g., hosts/GO_0xxx/, hosts/COISS_xxxx/, etc.

2. Copy the python files from an existing host directory and rename them as needed for the new collection.

3. Edit the config and init modules as needed.

4. Create a templates/ subdirectory and copy the files from an existing host. Rename the files as needed.

5. Edit host_defs.lbl for the new host.

6. Edit the descriptions in the summary templates as needed.

7. Edit the supplemental template to define the supplemental metadata for the new host.

Modifying table columns

The supplemental index table is controlled by the supplemental label template. By default, each column object in the label specifies the name of a PDS label field to add to the table, along with its desired formatting. This behavior may be overridden by adding a key function to index_config.py of the form:

key__<NAME>(label_path, label_dict)

where label_path is the path to the PDS label, and label_dict is a dictionary containing the PDS label fields. The returned value is placed in the table.

Modifying the geometry tables requires editing of the column definition and format tables, and may require the addition of new backplane functions.

To add a new geometry column:

1. Add a column definition to a column definition file, e.g. columns/body.py.

2. Add a corresponding function to appropriate backplane module.

3. Add a row to the format dictionary in geometry_support.py.

4. Add column description(s) to the label template, e.g., body_summary.lbl.

metadata_tools.common

Common classes and functions shared across the metadata tools.

Provides the global PdsLogger, cloud-task plumbing, the shared argument parser, and the Table base class used by the index and geometry table generators.

class PathAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: Action

argparse action that normalizes redundant slashes in path arguments.

__call__(parser: ArgumentParser, namespace: Namespace, values: Any, _option_string: str | None = None) → None

Normalize the path value and store it on the namespace.

Parameters:

• parser – The argument parser invoking this action.

• namespace – Namespace on which to store the normalized value.

• values – The raw argument value, or a list whose first element is used.

• _option_string – The option string that triggered this action, if any.

class Table(output_dir: str | Path | FCPath | None = None, template_path: str | Path | FCPath | None = None, volume_id: str | None = None, level: str | None = None, qualifier: str | None = None, prefix: str | None = None, suffix: str | None = None, use_global_template: bool = False)

Bases: object

Base class describing a single metadata table for a single volume.

write(labels_only: bool = False) → None

Write a table and its label.

Parameters:

labels_only – If True, labels are generated for any existing geometry tables.

add_task(volume_id: str, index_type: str) → None

Add a task to the task list.

Parameters:

• volume_id – ID of volume to add.

• index_type – ‘index’ or ‘geometry’.

get_common_args(host: str | None = None, volume_arg: str | None = 'volume_tree', metadata_arg: str | None = 'metadata_tree', output_arg: str | None = 'output_tree') → ArgumentParser

Common argument parser for metadata tools.

Parameters:

• host – Host name, e.g. ‘GOISS’.

• volume_arg – Name of volume_tree arg or None to skip this argument.

• metadata_arg – Name of metadata_tree arg or None to skip this argument.

• output_arg – Name of output_tree arg or None to skip this argument.

Returns:

Parser containing the common argument specifications.

get_logger() → PdsLogger

The global PdsLogger for the metadata tools.

init_logger(log_dir: FCPath, log_type: str) → None

Initialize the global logger with file and stdout handlers.

Parameters:

• log_dir – Directory to log.

• log_type – Type of log to create.

task_source() → Iterator[dict[str, Any]]

Task source generator for cloud_tasks.

Yields:

Each task dictionary from the accumulated task list.

write_task_file(task_file: str | None) → None

Write the tasks file.

Parameters:

task_file – Name of file to write.

metadata_tools.label_support

Tools for generating PDS3 metadata labels from templates.

create(filepath: str | Path | FCPath, host_template_path: str | Path | FCPath | None, system: str | None = None, *, use_global_template: bool = False, table_type: str | None = '') → None

Create a label for a given geometry table.

Parameters:

• filepath – Path to the local or remote table.

• host_template_path – Path to the host template.

• system – Name of system, for rings and moons.

• use_global_template – If True, the label template is to be found in the global template directory.

• table_type – BODY, RING, SKY, SUPPLEMENTAL_INDEX, INVENTORY.

metadata_tools.util

Utility functions for path handling, file I/O, and metadata computations.

add_by_base(x_digits: list[int], y_digits: list[int], bases: list[int]) → list[int]

Add numbers represented using the specified bases.

Parameters:

• x_digits – Digits (int) representing the first operand.

• y_digits – Digits (int) representing the second operand.

• bases – Bases (int) for each position.

Returns:

Digits (int) representing the result.

append_txt_file(filespec: str | Path | FCPath, content: str | list[str], terminator: str | None = '\r\n') → None

Append text to a file, with some options.

Parameters:

• filespec – Path to the file to write.

• content – Text to write. If list, each element is a line that will be terminated using the specified terminator. If string, existing terminators are replaced with the specified terminator.

• terminator – Desired line terminator.

convert_mission_table(table: list[Any], sc: int) → list[Any]

Convert mission table SCLK count string to ticks using sclk_to_ticks().

Parameters:

• table – Systems table.

• sc – NAIF spacecraft identifier.

Returns:

Converted mission table containing ticks instead of strings.

expandvars(filespec: str | Path | FCPath) → str | Path | FCPath

Expand environment variables in path.

Parameters:

filespec – Path to expand.

Returns:

Expanded path.

get_index_name(tree: FCPath, vol_id: str, index_type: str) → str

Determine the name of the index file.

Parameters:

• tree – Top dir for volume.

• vol_id – Volume ID.

• index_type – Index type.

Returns:

Index name.

get_observation_id(observation: Any) → str

Determine the observation ID for an observation.

Parameters:

observation – Observation object.

Returns:

Observation ID.

get_template_name(filename: str, volume_id: str, code_dir: FCPath) → str

Determine the name of the label template.

Parameters:

• filename – Name of table or label file.

• volume_id – Volume ID to be replaced by the collection name.

• code_dir – Directory whose name is the collection name.

Returns:

Index name.

get_volume_glob(col: str) → str

Build a glob string to match all volumes in a collection.

Parameters:

col – Collection name, e.g., GO_xxxx.

Returns:

Glob string.

get_volume_subdir(path: FCPath, volume_id: str) → FCPath

Determine the subdirectory of an input file relative to the volume dir.

Parameters:

• path – Input path or directory.

• volume_id – Volume ID at which to split the path.

Returns:

Final directory in the tree.

parse_template_name(template_name: str) → tuple[str, str, FCPath]

Determine host and index type from template name of the form:

<host>_<index type>_index

index_type cannot contain underscores.

Parameters:

template_name – Name of template file.

Returns:

Tuple of host name, index type, and the template directory.

pds_table(label_path: FCPath) → PdsTable

Read a PdsTable from an FCPath object.

Parameters:

label_path – Path to the label file.

Returns:

Table associated with the given label.

pm(x: float) → ndarray[tuple[Any, ...], dtype[float64]]

Return plus/minus the input.

Parameters:

x – Value to negate.

Returns:

Plus and minus values.

range_of_n_angles(n: int, prob: float = 0.1, tests: int = 100000) → float64

Study the statistics of n randomly chosen angles.

Used to compute the NINETY_PERCENT_RANGE_DEGREES table below. For a set of n randomly chosen angles 0-360, return the interval such that the likelihood of all n angles falling within this interval of one another has the given probability. Base this on the specified number of tests.

Parameters:

• n – Number of random samples to analyze.

• prob – Probability criterion.

• tests – Number of tests to perform.

Returns:

Angular interval in degrees.

read_txt_file(filespec: str | Path | FCPath, as_string: bool = False, terminator: str = '\r\n') → str | list[str]

Read a text file, with some options.

Parameters:

• filespec – Path to the file to read. Environment variables are expanded.

• as_string – If True, the result is returned as a string using the specified terminator.

• terminator – Terminator to use for string return.

Returns:

If as_string is False, the lines of the file with no terminators; if True, the lines of the file concatenated using the specified terminator.

rebase(x: int, bases: list[int], ceil: bool = False) → tuple[list[int], int]

Convert a decimal number to a different base.

Parameters:

• x – Number to convert.

• bases – Base (int) to use for each decimal place.

• ceil – If True, round each digit up rather than truncating.

Returns:

Tuple of the digits (int) in the new base and the remaining quantity, if any, exceeding the maximum value that can be represented by the given bases.

replace(tree: list[Any], placeholder: str, name: str) → Any

Return a copy of the tree of objects, with each occurrence of the placeholder string replaced by the given name. If a dictionary reference is detected, then it is evaluated.

Parameters:

• tree – List containing the tree.

• placeholder – Placeholder to replace.

• name – Replacement string.

Returns:

New tree with placeholder replaced by name.

replacement_dict(tree: list[Any], placeholder: str, names: list[str]) → dict[str, Any]

Create a dictionary of copies of the tree of objects, where each dictionary entry is keyed by a name in the list and returns a copy of the tree using that name as the replacement.

Parameters:

• tree – List containing the tree.

• placeholder – Placeholder to replace.

• names – List of replacement strings.

Returns:

New dictionary.

replacement_fn(dict_name: str, name: str) → str

Create a replacement-able dictionary reference.

Parameters:

• dict_name – Name of dictionary.

• name – Dictionary key, which could be a placeholder string.

Returns:

Dictionary reference keyed by possible placeholder name.

sclk_format_count(fields: list[int], fmt: str) → str

Construct a spacecraft clock count from a list of fields.

Parameters:

• fields – Fields (int) of the spacecraft clock count.

• fmt – Template indicating the field widths and delimiters. Alphanumeric characters indicate field digits, non-alphanumeric characters indicate field delimiters. Example: ‘nnnnnnnn:nn:n.n’.

Returns:

Spacecraft clock count, zero-padded and delimited per the format.

sclk_split_count(count: str, delim: str | None = None) → list[int]

Parse a spacecraft clock count into a list.

Parameters:

• count – Number to convert.

• delim – Field delimiter to use. If None, all non-alphanumeric characters are treated as delimiters.

Returns:

Fields (int) of the given spacecraft clock count.

sclk_to_ticks(sclk: str, sc: int) → Any

Convert spacecraft clock count string to ticks.

Parameters:

• sclk – Spacecraft clock count string.

• sc – NAIF spacecraft identifier.

Returns:

Spacecraft clock ticks.

select_dir(tree: FCPath, col: str, vol: str) → FCPath

Determine the template directory for a given collection and volume.

Parameters:

• tree – Base tree path.

• col – Collection name.

• vol – Volume name.

Returns:

Directory path.

smooth(data: ndarray[tuple[Any, ...], dtype[float64]], width: int) → ndarray[tuple[Any, ...], dtype[float64]]

Smooth a curve using a moving box.

Parameters:

• data – Data to smooth.

• width – Width of smoothing box.

Returns:

Smoothed data.

splitpath(path: FCPath, string: str) → tuple[FCPath, FCPath]

Split a path at a given string.

Parameters:

• path – Path to split.

• string – Search string. The path is split at the first occurrence and the search string is omitted.

Returns:

Tuple of the path portion before the search string and the portion after it.

write_txt_file(filespec: str | Path | FCPath, content: str | list[str], terminator: str | None = '\r\n') → None

Write a text file, with some options.

Parameters:

• filespec – Path to the file to write.

• content – Text to write. If list, each element is a line that will be terminated using the specified terminator. If string, existing terminators are replaced with the specified terminator.

• terminator – Desired line terminator.

metadata_tools.defs

Shared constants for the metadata tools.

Defines package paths, null markers, body-name lists, and ring-system radii used throughout the index and geometry table generators.

BODYX = 'bodyx'

Placeholder body name, substituted per body by the column-definition replacement helpers (see metadata_tools.util.replacement_dict()).

BODY_NAMES = ['MERCURY', 'VENUS', 'EARTH', 'MARS', 'JUPITER', 'SATURN', 'URANUS', 'NEPTUNE', 'PLUTO']

Planet names whose systems are processed, in registry order.

GLOBAL_TEMPLATE_PATH = FCPath('/home/docs/checkouts/readthedocs.org/user_builds/rms-metadata-tools/checkouts/latest/src/metadata_tools/templates')

Directory holding the package’s shared PDS3 label-template fragments.

NAME_LENGTH = 12

Character width of a column that holds a body name.

NULL = 'null'

Null marker used in column definitions to denote an absent value.

PARENT_DIR = FCPath('/home/docs/checkouts/readthedocs.org/user_builds/rms-metadata-tools/checkouts/latest/src/metadata_tools')

Directory containing the installed package.

RING_SYSTEM_RADII = {'EARTH': 0, 'JUPITER': 128940.0, 'MARS': 0, 'MERCURY': 0, 'NEPTUNE': 62940.0, 'PLUTO': 0, 'SATURN': 136780.0, 'URANUS': 51604.0, 'VENUS': 0}

Outer radius (km) of each planet’s main ring system; 0 if it has none.

TRANSLATIONS: dict[str, str] = {}

Optional mapping of raw target names to canonical body names.

metadata_tools.bodies

Build the oops Body registry used by the geometry column tables.

BODIES = {'EARTH': <_FakeBody object>, 'JUPITER': <_FakeBody object>, 'MARS': <_FakeBody object>, 'MERCURY': <_FakeBody object>, 'NEPTUNE': <_FakeBody object>, 'PLUTO': <_FakeBody object>, 'SATURN': <_FakeBody object>, 'URANUS': <_FakeBody object>, 'VENUS': <_FakeBody object>}

Mapping from body name to its oops Body object, built once on import.

get_bodies(body_names: list[str]) → dict[str, Any]

Build a registry mapping each body name to its oops Body object.

For every name, the body itself and its regular children are included.

Parameters:

body_names – Names of the primary bodies to look up.

Returns:

Mapping from body name to the corresponding oops Body object, including each primary body’s regular children.