induction-airflow

Induction to Airflow

View the Project on GitHub infra-helpers/induction-airflow

Induction to Workflow Engines - Airflow

Canyon, by Patrick Hendry on Unsplash

Overview

That repository aims at providing end-to-end examples introducing how to setup, run and monitor workflows, mainly for data processing pipelines.

In those tutorials, Airflow is used. A full end-to-end example is explained step by step, and actually used for the Open Travel Data (OPTD) project.

The full details on how to setup an Airflow server on Proxmox LXC containers are given in the dedicated airflow/ sub-folder. Such an Airflow cluster is actually the orchestration engine of the Open Travel Data (OPTD) project.

For convenience, most of the Airflow examples are demonstrated both on a local single-node installation (e.g., on a laptop) and on on the above-mentioned cluster.

Endpoints

Table of Content (ToC)

Table of contents generated with markdown-toc

References

Configuration