Kafka Producer

1️⃣ Checking Directories and Kafka Availability

Navigate to the home directory and list all elements. Check if the kafka directory is present in the list.

cd ~
ls -la

2️⃣ Checking the List of Topics

Run the following command to check the list of topics on the Kafka server:

kafka/bin/kafka-topics.sh --list --bootstrap-server broker:9092

3️⃣ Creating a New Topic

Create a topic named streaming:

kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic streaming

Check the list of topics again to ensure that streaming has been added:

kafka/bin/kafka-topics.sh --list --bootstrap-server broker:9092 | grep streaming

4️⃣ Creating a New Python Producer script

In a new terminal, create a file named stream.py and paste the following code:

%%file stream.py
import json
import random
import sys
from datetime import datetime, timedelta
from time import sleep

from kafka import KafkaProducer

SERVER = "broker:9092"
TOPIC = "streaming"

if __name__ == "__main__":
    
    
    producer = KafkaProducer(
        bootstrap_servers=[SERVER],
        value_serializer=lambda x: json.dumps(x).encode("utf-8")
    )
    
    try:
        while True:
            
            message = {
                # Your Code Here
            }
            producer.send(TOPIC, value=message)
            sleep(1)
    except KeyboardInterrupt:
        producer.close()

Writing stream.py

Running a Console Consumer

To check if message sending works, open another terminal window and start the consumer:

kafka/bin/kafka-console-consumer.sh --bootstrap-server broker:9092 --topic streaming --from-beginning

Now, all messages sent by the producer should appear in the consumer console.

Finishing Up

Remember to run commands from the appropriate directory. When you’re done with the exercises, use Ctrl+C to stop both the producer and the consumer.

---

Now you have a basic understanding of how Apache Kafka works 🚀