Data Imports in the DataLab

The datalab-tools application has the ability to import data directly into the Aershed Platform. The datalab-tools application has a wrapper over the import client from the data_piplines respositry, which is imported as a submodule.

By managing imports directly in the datalab-tools application we can add additional protections and consistency checks to data prior to importing. The most crucial is the ability to read data from the Aershed database within the import task, which is something we can only do if we combine both ingest and import code into this one application. For example, we can check if the data that is about to be imported already exists, and stop the import.

Other important checks are performed to prevent an import happening twice, log data on what was imported and consistency checking between the original processing job configuration and the parameters used in the import.

Import metadata

Every import requires a metadata JSON file. Without this file the importer will ignore the folder and its contents. This file is used to communicate information between the processing job and the import, and any future execution of the import job. It is a JSON file on disk alongside the data to be processed, this does make it possible for a user to corrupt or delete this file so care must be taken and awareness that this system is not entirely safe.

The import metedata looks like the following after an import has started.

[
    {
        "owner": "Aerscape",
        "type": "bridger.emissions",
        "folder": "emissions",
        "processing_env": "prod-testing",
        "extra_data": {
            "survey": "BRIDGER_SURVEY_1",
            "detection_limit": 3000,
            "unique_id_col": "record_id"
        },
        "last_updated": "2026-07-20 06:44:14",
        "import_started_time": "2026-07-20 06:39:36",
        "import_id": "9e4f2d17-4a2a-4bee-a8f7-834a6d0bbd26",
        "status": "IN_PROGRESS"
    },
    {
        "owner": "Aerscape",
        "type": "bridger.non_detections",
        "folder": "site_non_detections",
        "processing_env": "prod-testing",
        "extra_data": {
            "survey": "BRIDGER_SURVEY_1",
            "detection_limit": 3000
        },
        "last_updated": "2026-07-20 07:55:11",
        "import_id": "540d26fb-d3b0-44fd-8575-81392618e7ff",
        "status": "COMPLETE"
    }
]

In this case the folder that this JSON file is in has two elegible assets to import. This metadata tells the importer what kind of imports these are and where to find them. Once an import has stated the importer writes to this file to update the status, the import_id and other relevant information.

So in order to create a file for import you must also supply an import metadata file. This file is then used to track the import, prevent the import being started when it is already in progress and to pass important information between the processing job and the import - such as survey name and the operator code.

Some processing jobs will not run if the import metadata shows the import as being in an in progress or final state. This is to prevent overwritting the data that has already been sent to Aershed. If you need to reprocess the data then you can delete the import metadata file or point your job at a clean directory.

Depending on the existance of an import metadata and persistence of the entire output directory should not be the only way to prevent duplication of data in the platform. It is one tool to avoid mistakes but it is not the only protection. All jobs should have some level of duplication checks either in the processing or import tasks, or both.

Import statuses

Each import is in one of these states:

• In-progress: CREATED, READY_FOR_PROCESSING, IMPORTING, WAITING_APPROVAL — while an import is in any of these, its data counts toward overlap checks.
• Final: ERROR, COMPLETE, TIMED_OUT — once an import reaches one of these, it's no longer relevant to duplicate checking and is cleaned up.

Import Tracker

Imports into the Aershed Platform take time to process. An import goes through several stages before it's finished, and until it reaches one of those final stages we cannot easily check the next import for potential duplicates. This creates a risk that the next import task to run could submit the same data a second time while the first import is still in progress.

The import tracker keeps its own local record of every import that's currently in progress in a local sqlite database shared by all users on the same datalab instance (and scoped per Aershed environment). Before a new import is allowed to start, the system checks this local record to see whether the new data overlaps with anything already pending.

Which imports can run in parallel

Some import types are limited to one in-progress import at a time per operator (owner) — if one is already running, any new import of that same type is blocked until the first one finishes. Currently on infrastructure imporst are limited to one at a time. This is because the infrastructure processing job is always a difference with respect to the current state.

All other import types are allowed to have more than one in progress at once, because the system is able to check their data for overlap directly instead of just blocking on type alone. These are:

• Emissions data: Bridger Emissions, GHGSat Emissions, GHGSat Global Emissions, Generic Emissions — checked for overlap by matching record IDs.
• Non-detection data: Bridger Non-Detections, GHGSat Non-Detections, Generic Site Non-Detections — checked for overlap by matching site and date together.

For these types, a new import isn't blocked just because another is running — its data is compared row-by-row against what is already pending, and it is only blocked if an actual overlap is found.