#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-ingestion/building-custom-connectors

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░███████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/data-ingestion/building-custom-connectors

Repovive

Building Custom Connectors - Data Ingestion | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Building Custom Connectors

When you need to roll your own

Custom connectors make sense for:

Internal APIs with no managed support
Unusual data formats or protocols
Cost optimization at scale

Key considerations:

Rate limiting: Respect API limits. Implement exponential backoff.
Incremental loads: Track watermarks to avoid full syncs.
Error handling: Log failures, retry transient errors, alert on persistent ones.
Schema changes: Detect and handle upstream changes gracefully.

def sync_api(last_sync_time):
    records = fetch_since(last_sync_time)
    write_to_staging(records)
    update_watermark(max_timestamp(records))