#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/streaming-and-real-time-data/kafka-partitions-and-consumer-groups

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/streaming-and-real-time-data/kafka-partitions-and-consumer-groups

Repovive

Kafka Partitions and Consumer Groups - Streaming and Real-Time Data | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Kafka Partitions and Consumer Groups

Scaling consumption

Partitions split a topic across brokers. A topic with $12$ partitions can have data on $12$ brokers.

Ordering is guaranteed only within a partition. Events with the same key go to the same partition.

Consumer Groups enable parallel consumption. Each partition is read by exactly one consumer in the group.

Topic: orders (6 partitions)
Consumer Group: order-processor

Consumer 1: partitions 0, 1
Consumer 2: partitions 2, 3
Consumer 3: partitions 4, 5

Add consumers to increase throughput. But you can't have more consumers than partitions in a group.