call_split

Experiment 3: Partitions & Consumer Groups

Topic Sharding · Parallel Consumption · Rebalancing

arrow_back Back to Experiments
Scalability & Sharding

Scale with Partitions

By adding partitions to a topic, Kafka enables parallel processing. Multiple consumers in the same Consumer Group will automatically shard the load.

flag Aim

Create a topic with 2 partitions and start 2 consumers in the same group. Watch how Kafka balances the load, sending half the messages to Consumer A and half to Consumer B.

route Theory: Partitions & Consumer Groups

To scale processing, Kafka splits topics into Partitions. When multiple consumers join the same Consumer Group, Kafka automatically assigns each partition to exactly one consumer, distributing the workload evenly.

Topic
Partition 0
Partition 1
arrow_right_alt arrow_right_alt
Consumer Group
download Consumer A
download Consumer B

checklist Mission Milestones

0 / 4 Done
radio_button_unchecked 1. Broker Online
radio_button_unchecked 2. Create Topic (2 Partitions)
radio_button_unchecked 3. Consumer A Subscribed
radio_button_unchecked 4. Consumer B Subscribed

format_list_numbered Commands — Follow Order!

1

Create Topic (Terminal 2)

kafka-topics.bat --create --topic fsd-topic --partitions 2 --bootstrap-server 192.168.1.10:9092

Make sure the broker is running first (click Auto-Boot in Terminal 1).

2

Start Consumer A (Terminal 3)

kafka-console-consumer.bat --topic fsd-topic --group app-group --bootstrap-server 192.168.1.10:9092
3

Start Consumer B (Terminal 4)

kafka-console-consumer.bat --topic fsd-topic --group app-group --bootstrap-server 192.168.1.10:9092
4

Send Messages (Terminal 2)

kafka-console-producer.bat --topic fsd-topic --bootstrap-server 192.168.1.10:9092

Type 4 different messages. Watch how they are distributed!

lightbulb

Messages typed in Terminal 2 will appear instantly in Terminal 3 once the consumer is running!

Terminal 1 — Broker (192.168.1.10)
Offline
KRaft Broker — Ready to initialize.
C:\kafka4.3.1>
Terminal 2 — Producer
Producer
Remote Kafka Producer Node on LAN.
C:\kafka4.3.1>
Terminal 3 — Consumer A (192.168.1.30)
Consumer A
Consumer Node A in group 'app-group'.
C:\kafka4.3.1>
Terminal 4 — Consumer B (192.168.1.40)
Consumer B
Consumer Node B in group 'app-group'.
C:\kafka4.3.1>