The Topo Processor is a collection of small components that can be combined together to create a pipeline.
These components include transforming data into cloud optimised formats like COG and the creation of STAC metadata.
Follow the Poetry installation guide.
Follow the Docker Engine installation guide (Ubuntu).
poetry shell
poetry install# Run in a virtual environment (poetry shell):
./upload --source source_path --datatype data.type --target target_folder# For help:
./upload --help# To see all logs in a tidy format, use pretty-json-log:
./upload --source source_path --datatype data.type --target target_folder --verbose | pjlThe following source and target combinations can be used:
| Source | Target |
|---|---|
| s3 | s3 |
| s3 | local |
| local | local |
| local | s3 |
This command runs a validation against a metadata csv file. It generates the corresponding STAC objects on the fly for each metadata and run a JSON schema validation (using jsonschema-rs) for the Items and Collections. It outputs the errors and their recurrences grouped by JSON schemas as:
"errors": {"https://linz.github.io/stac/v0.0.7/aerial-photo/schema.json": {"'aerial-photo:run' is a required property": 4, "'aerial-photo:sequence_number' is a required property": 10}# Run in a virtual environment (poetry shell):
./validate --source metadata_file.csv# Run against the `Items` only:
./validate --source metadata_file.csv --item# Run against the `Collections` only:
./validate --source metadata_file.csv --collection# For help:
./validate --help# To see all logs in a tidy format, use pretty-json-log:
./validate --source metadata_file.csv --verbose | pjl# To record the output in an external file:
./validate --source metadata_file.csv | tee output.file