Experiment 5: Python Streaming

Application Development

Kafka meets Python

CLI tools are great for debugging, but real-world event streaming happens in code. In this lab, you will write a programmatic Producer and Consumer using the kafka-python library.

flag Aim

To transition from shell scripts to application development. You will learn how to initialize connection properties, construct message payloads, and write iteration loops to fetch records dynamically in Python. This is exactly how production systems ingest data into databases or machine learning pipelines.

account_tree Architecture: Python Integration

code Python App
(Producer)

arrow_right_alt

Kafka Broker

Topic: fsd-topic

arrow_right_alt

terminal Python App
(Consumer)

Instead of relying on the built-in Java CLI, your Python applications communicate directly with the Kafka broker over TCP using the Kafka wire protocol.

data_object Theory: The python-kafka API

The kafka-python library provides two main classes that abstract away the complex network communication:

1. Producer API

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['192.168.1.10:9092'])
producer.send('fsd-topic', b'Hello from Python!')

2. Consumer API

from kafka import KafkaConsumer

consumer = KafkaConsumer('fsd-topic', bootstrap_servers=['192.168.1.10:9092'])
for msg in consumer:
print(msg.value)

checklist Mission Milestones

0 / 4 Done

radio_button_unchecked 1. Start the Kafka Broker

radio_button_unchecked 2. Write and execute Producer script

radio_button_unchecked 3. Write and execute Consumer script

radio_button_unchecked 4. Modify Producer to send exactly 15 messages (Hint: change range(10) to range(15))

warning

This IDE simulator features Strict Syntax Validation. If you miss a colon, forget a parenthesis, or mess up indentation in your Python code, it will fail to run!

Terminal 1 — Broker (192.168.1.10)

Offline

KRaft Broker — Ready to initialize.

1
2
3
4
5
6
7
8
9
10
11
12

terminal Console Output

Awaiting execution...

monitoring Live Chart

Run consumer to see
live JSON telemetry