#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-pipeline-orchestration/backfills-and-catchup

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░███████████████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-pipeline-orchestration/backfills-and-catchup

Repovive

Backfills and Catchup - Data Pipeline Orchestration | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Backfills and Catchup

Reprocessing historical data

Backfilling reprocesses historical data. Airflow supports this natively:

airflow dags backfill -s 2024-01-01 -e 2024-01-31 daily_etl

Catchup: When catchup=True (default), Airflow runs all missed intervals when a DAG starts. Set catchup=False if you only want future runs.

Design for backfills: Your tasks should accept a date parameter and process only that date's data. Avoid hardcoded "yesterday" logic.

Backfill-friendly pipelines save hours of debugging later.