Distributed (cloud) runs

Generating geometry for a large collection is CPU-bound and embarrassingly parallel across volumes. Each host therefore ships *_cloud.py counterparts to the three programs (<HOST>_index_cloud.py, <HOST>_geometry_cloud.py, <HOST>_cumulative_cloud.py) that distribute the per-volume work using the rms-cloud-tasks framework on Google Cloud Platform (GCP).

These workers require the cloud optional dependencies:

pip install -e ".[cloud]"

How it works

The cloud programs reuse the same engine entry points (process_index(), process_tables()) in two modes:

  1. Build a task list. Running the program with --task-output (or letting the worker build it) produces a task file: one task per volume. No tables are generated in this mode.

  2. Process tasks. A worker pool consumes the task file, calling the engine once per volume with the volume ID from each task.

Local parallel runs

Run a cloud program exactly like its plain counterpart, with additional rms-cloud-tasks worker options such as --num-simultaneous-tasks:

cd src/metadata_tools/hosts/GO_0xxx
python GO_0xxx_index_cloud.py "$RMS_VOLUMES/GO_0xxx/" "$RMS_METADATA/GO_0xxx/" \
    "$RMS_METADATA_TEST/GO_0xxx/" --num-simultaneous-tasks 12

GCP runs

For a run on GCP, authenticate, generate the task file, and submit it with the host’s paired configuration:

gcloud auth application-default login        # if necessary

python GO_0xxx_index_cloud.py "$RMS_VOLUMES/GO_0xxx/" "$RMS_METADATA/GO_0xxx/" \
    "$RMS_METADATA_TEST/GO_0xxx/" -to index_tasks.json
cloud_tasks run --config gcp_index_config.yml --task-file index_tasks.json --use-spot

Each host directory contains the gcp_*_config.yml machine/queue configuration and the gcp_*_startup.sh instance start-up script referenced above.

Task file schema

The task file is JSON: a list of task objects, one per volume. Each object has a unique task_id and a data payload carrying the volume ID that the worker passes back to the engine:

[
  {
    "task_id": "geometry-task-GO_0017",
    "data": { "volume_id": "GO_0017" }
  },
  {
    "task_id": "geometry-task-GO_0018",
    "data": { "volume_id": "GO_0018" }
  }
]

The task_id prefix identifies the stage that produced the file. The worker reads each entry, invokes the engine for data.volume_id, and reports success or failure back to rms-cloud-tasks.