Datalab tools framework

This package follows a structure that enables all programs to use a common configuration system and make use of other useful services via interfaces.

The program structure

Each task is defined as a program with at least one argument that takes a configuration object. The original and simplest form is the following.

def example(config: BaseConfig) -> None:
    logger.info("Hello from example program")

This design quickly become too limited so we now have a more advanced option using a function decorator to populate additional services while still being called from the same entry point. For example the following pattern can be used.

@inject
def run(
    config: BaseConfig,
    writer: IFileSystem,
    file_loader: IFileLoader,
    data_importer_client: IDataImportClient,
    database_client: IPlatformDB,
):

In this case the decorator inject automatically creates the required implementations of each service, calling their respective factory methods. This reduces the need for each program to repeat the steps to create these interfaces and enables us to inject mock implementations for testing or when running against a local environment. It means you can develop using a service without worrying about how it is created.

The current list of available services is given below. For the most up to date list of what is available, and to see and modify factory methods, see the file "datalab_tools/core/inject.py".

• IFileSystem - used for writing to output files on disk;
• IFileLoader - for reading files from local drive, S3 or from external sources (such as Box);
• IDataImportClient - a wrapper to the Aershed import client for import related tasks;
• IPlatformDB - an interface to the Aershed database for read-only operations;
• ISlackClient - for sending Slack messages to the datalab_automation channel;
• IImportTracker - for creating or updating imports and checking if imports are already in progress using a local sqlite database;
• IGHGSatClient - a wrapper around the GHGSat API for fetching data.

It is straight forward to add a new service by following the existing patterns. Not all code must be a service, but if you are creating something that would need to be mocked in a test and shared across multiple other programs then following this service framework can be useful.

Environment scoping

See the working with Aershed documentation for more details on how environment scoping works. Anything not related to Aershed (requiring data or making imports) can be done with the local environment, using the --local command line argument. For the local environment the processing directory will be ${PROCESSING_DIR}/local where PROCESSING_DIR is the value you set in your base environment file.

Exception reporting

Programs are wrapped in an exception handler which stores the original exception and re-raises it with information on the program that was running at the time. This exception is then converted into a Slack message if reporting is enabled.

To enable Slack message reporting to the datalab-automation channel the program must be called with the --report_exceptions argument. This cannot be provided in the configuration file and must be a command line argument.

Reporting is only enabled in those environments listed in the AERSHED_ENVIRONMENTS_NOTIFY_SLACK and for those which have access to the Slack hook URL environment variable (SLACK_DATA_IMPORT_HOOK_URL).