#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/performance-optimization/incremental-processing

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░█████████████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/performance-optimization/incremental-processing

Repovive

Incremental Processing - Performance Optimization | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Incremental Processing

Process only what changed

Reprocessing everything daily wastes resources. Incremental processing handles only new or changed data.

Patterns:

$1.$ Append-only: New records added to existing table. Works for immutable event data.

$2.$ Merge/Upsert: Update existing records, insert new ones. Requires unique key.

$3.$ Partition replacement: Reprocess only affected partitions.

dbt incremental models:

{{ config(materialized='incremental') }}

SELECT * FROM {{ source('raw', 'events') }}
{% if is_incremental() %}
WHERE event_time > (SELECT MAX(event_time) FROM {{ this }})
{% endif %}

This reduces runtime from hours to minutes.