#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-pipeline-orchestration/apache-airflow-core-concepts

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░██████████████████████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-pipeline-orchestration/apache-airflow-core-concepts

Repovive

Apache Airflow Core Concepts - Data Pipeline Orchestration | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Apache Airflow Core Concepts

DAGs and tasks

Airflow organizes work into Directed Acyclic Graphs (DAGs). Each DAG contains tasks with dependencies:

from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG('daily_etl', schedule='@daily') as dag:
    extract = PythonOperator(task_id='extract', ...)
    transform = PythonOperator(task_id='transform', ...)
    load = PythonOperator(task_id='load', ...)
    
    extract >> transform >> load

DAG: The workflow definition. No cycles allowed (A can't depend on B if B depends on A).

Task: A single unit of work. Tasks are instances of Operators.