Data Imports in the DataLab

The datalab-tools application has the ability to import data directly into the Aershed Platform. It uses a wrapper over the import client from the data_pipelines repository, which is imported as a submodule.

By managing imports directly in this application we can add additional protections and consistency checks to data prior to importing. The most crucial is the ability to read data from the Aershed database within the import task, which is something we can only do if we combine both ingest and import code into this one application. For example, we can check if the data that is about to be imported already exists, and stop the import.

Other important checks are performed to prevent an import happening twice, log data on what was imported and consistency checking between the original processing job configuration and the parameters used in the import.

Import command line options

Import tasks are guarded to avoid running them accidentally. To run any import tasks the job must be called with the required flags. This enables us to run processing jobs as many times as we like prior to import and inspect the import files without having to comment out the import task. Alternatively we would have to have different configuration files for the processing and import jobs, which can lead to inconsistencies. This approach lets us keep all the steps in one place.

We have two command line flags to enable imports to run.

• --import - this will run all the tasks in the pipeline including any imports;
• --import_only - this will only run the import tasks and skip any other tasks in the pipeline.

The --import_only flag is very useful, for most jobs that we are doing manually it makes sense to run the pipeline without imports enabled first, check the results and then rerun with this flag. Being able to only run import tasks saves reprocessing everything again. This of course only makes sense if your pipeline is laid out in order of data processing, then imports.

For automated jobs we would only use the --import flag.

Any program can be protected with the import guard by adding the @import_guard() decorator above the run function definition.

Import metadata

Every import requires a metadata JSON file. Without this file the importer will ignore the folder and its contents. This file is used to communicate information between the processing job and the import, and any future execution of the import job. It is a JSON file on disk alongside the data to be processed, this does make it possible for a user to corrupt or delete this file so care must be taken and awareness that this system is not entirely safe.

The import metadata looks like the following after an import has started.

[
    {
        "owner": "Aerscape",
        "type": "bridger.emissions",
        "folder": "emissions",
        "processing_env": "prod-testing",
        "extra_data": {
            "survey": "BRIDGER_SURVEY_1",
            "detection_limit": 3000,
            "unique_id_col": "record_id"
        },
        "last_updated": "2026-07-20 06:44:14",
        "import_started_time": "2026-07-20 06:39:36",
        "import_id": "9e4f2d17-4a2a-4bee-a8f7-834a6d0bbd26",
        "status": "IN_PROGRESS"
    },
    {
        "owner": "Aerscape",
        "type": "bridger.non_detections",
        "folder": "site_non_detections",
        "processing_env": "prod-testing",
        "extra_data": {
            "survey": "BRIDGER_SURVEY_1",
            "detection_limit": 3000
        },
        "last_updated": "2026-07-20 07:55:11",
        "import_id": "540d26fb-d3b0-44fd-8575-81392618e7ff",
        "status": "COMPLETE"
    }
]

In this case the folder that this JSON file is in has two eligible assets to import. This metadata tells the importer what kind of imports these are and where to find them. Once an import has stated the importer writes to this file to update the status, the import_id and other relevant information.

So in order to create a file for import you must also supply an import metadata file. This file is then used to track the import, prevent the import being started when it is already in progress and to pass important information between the processing job and the import - such as survey name and the operator code.

Some processing jobs will not run if the import metadata shows the import as being in an in progress or final state. This is to prevent overwriting the data that has already been sent to Aershed. If you need to reprocess the data then you can delete the import metadata file or point your job at a clean directory.

Depending on the existence of an import metadata and persistence of the entire output directory should not be the only way to prevent duplication of data in the platform. It is one tool to avoid mistakes but it is not the only protection. All jobs should have some level of duplication checks either in the processing or import tasks, or both.

Import statuses

Each import is in one of these states:

• In-progress: CREATED, READY_FOR_PROCESSING, IMPORTING, WAITING_APPROVAL — while an import is in any of these, its data counts toward overlap checks.
• Final: ERROR, COMPLETE, TIMED_OUT, ERROR_NOT_FOUND — once an import reaches one of these, it's no longer relevant to duplicate checking and is cleaned up.

The ERROR state will remain in the import tracker and needs to be cleared out using the following command.

./scripts/run.sh config/reset_import_tracker.yaml

The ERROR_NOT_FOUND state is a special case defined in the datalab-tools. It is used for the particular situation where an environment has been reset (typically prod-testing is reset using data from production) such that import histories specific to prod-testing are lost. So when a import status is queried, based on the import ID, the API will return a 404 error. Care should still be taken if this error is seen, the handling is designed to not obstruct operations against prod-testing but if seen elsewhere or when not expected further investigation may be required.

Import Tracker

Imports into the Aershed Platform take time to process. An import goes through several stages before it's finished, and until it reaches one of those final stages we cannot easily check the next import for potential duplicates. This creates a risk that the next import task to run could submit the same data a second time while the first import is still in progress.

The import tracker keeps its own local record of every import that's currently in progress in a local sqlite database shared by all users on the same datalab instance (and scoped per Aershed environment). Before a new import is allowed to start, the system checks this local record to see whether the new data overlaps with anything already pending.

Which imports can run in parallel

Some import types are limited to one in-progress import at a time per operator (owner) — if one is already running, any new import of that same type is blocked until the first one finishes. Currently on infrastructure imports are limited to one at a time. This is because the infrastructure processing job is always a difference with respect to the current state.

All other import types are allowed to have more than one in progress at once, because the system is able to check their data for overlap directly instead of just blocking on type alone. These are:

• Emissions data: Bridger Emissions, GHGSat Emissions, GHGSat Global Emissions, Generic Emissions — checked for overlap by matching record IDs.
• Non-detection data: Bridger Non-Detections, GHGSat Non-Detections, Generic Site Non-Detections — checked for overlap by matching site and date together.
• Aerial image data: Checked for overlap by checking for any overlapping bounding boxes on the same taken at date.

For these types, a new import isn't blocked just because another is running. Instead its data is compared row-by-row against what is already pending, and it is only blocked if an actual overlap is found.