Induction to Workflow Engines - Airflow

Overview

That repository aims at providing end-to-end examples introducing how to setup, run and monitor workflows, mainly for data processing pipelines.

In those tutorials, Airflow is used. A full end-to-end example is explained step by step, and actually used for the Open Travel Data (OPTD) project.

The full details on how to setup an Airflow server on Proxmox LXC containers are given in the dedicated airflow/ sub-folder. Such an Airflow cluster is actually the orchestration engine of the Open Travel Data (OPTD) project.

For convenience, most of the Airflow examples are demonstrated both on a local single-node installation (e.g., on a laptop) and on on the above-mentioned cluster.

Endpoints

• Airflow:
  • Local server: http://localhost:8080
  • Remote server: https://airflow.example.com

Table of Content (ToC)

• Induction to Workflow Engines - Airflow
• Overview
  • Endpoints
• Table of Content (ToC)
• References
• Configuration

Table of contents generated with markdown-toc

References

• Setup of a Proxmox host
• Setup of LXC containers on Proxmox
• Induction to Monitoring with Elasticsearch (ES), Kibana and Kafka

Configuration

• See airflow/