Kafka Virtual Lab - Experiment 4

Aim

Observe Leader Election

Start a 3-node Kafka cluster. Create a topic with a Replication Factor of 3. Then, crash the Leader Broker and watch how the remaining nodes elect a new leader without dropping data!

gpp_good Theory: High Availability (HA)

Kafka clusters provide fault tolerance through data replication. A topic with a Replication Factor of 3 stores copies of every message on 3 different brokers. One broker acts as the Leader, serving all read/write requests, while the other two are In-Sync Replicas (ISR). If the leader crashes, the cluster instantly promotes a new leader.

Producer

dns
Broker 1

dns
Broker 2

dns
Broker 3

Replication Factor = 3

checklist Mission Milestones

0 / 5 Done

radio_button_unchecked 1. All 3 Brokers Online

radio_button_unchecked 2. Replicated Topic Created (RF=3)

radio_button_unchecked 3. Producer Sent Messages

radio_button_unchecked 4. Leader Broker Crashed

radio_button_unchecked 5. New Leader Elected from ISR

format_list_numbered Commands

Create Topic (Terminal 4)

kafka-topics.bat --create --topic ha-topic --partitions 1 --replication-factor 3

Make sure all 3 brokers are running first (click Start Broker on all).

View Details (Terminal 4)

kafka-topics.bat --describe --topic ha-topic

Check the output to see who is the Leader and the ISR.

Send Messages (Terminal 4)

kafka-console-producer.bat --topic ha-topic

Crash the Leader

Look at the diagram to see which broker is the Leader. Click its Crash button.

View Details Again (Terminal 4)

kafka-topics.bat --describe --topic ha-topic

A new leader was elected! Notice the ISR list updated.

lightbulb

As long as one replica from the ISR is alive, your data is safe and the cluster stays available!

dns Node 1 (Broker 1)

Waiting for Broker 1 to start...

dns Node 2 (Broker 2)

Waiting for Broker 2 to start...

dns Node 3 (Broker 3)

Waiting for Broker 3 to start...

terminal Machine 4 (Client)

C:\kafka4.3.1>_

C:\kafka4.3.1>