Lecture 4: Apache Kafka and Microservices

Duration: 1.5h

Goal: Learn Apache Kafka’s architecture — a distributed streaming platform — and the concepts of microservices and APIs for serving data and ML models.

From Monolith to Microservices

Traditional IT systems were built as monoliths — a single large application where all functions (customer service, payments, reports, logging) are part of one codebase. Monoliths are simple at first, but quickly become hard to maintain — changing one module requires redeploying the whole thing.

The answer to these problems is microservices — an architecture where the system consists of many small, independent services, each handling one task and communicating with others via APIs.

Advantages of microservices:

  • Each service can be developed and deployed independently.
  • Failure of one service doesn’t bring down the entire system.
  • Easy scaling — add instances only of the services that need it.
  • Different technologies can be used in different services.

Challenges:

  • Communication complexity between services.
  • Need to monitor many components.
  • Managing distributed transactions.

In practice, microservices run in containers (Docker), managed by orchestrators (Kubernetes), and communicate via APIs (REST, gRPC) or asynchronous messaging systems (Apache Kafka).

REST API with FastAPI

An API (Application Programming Interface) is an interface through which services communicate with each other. In microservice architecture it’s the primary method of data exchange.

When you visit a website, your browser sends an HTTP request to a server, which returns an HTML page. An API works the same way — except instead of HTML it returns data (usually in JSON format).

Key REST API Principles

  • Client–Server — client sends a request, server returns a response.
  • Statelessness — each request contains all information needed to handle it. The server doesn’t remember previous requests.
  • HTTP Methods — GET (retrieve), POST (create), PUT (update), DELETE (remove).

Example: a simple microservice in FastAPI

from fastapi import FastAPI

app = FastAPI()

customers = {
    1: {"name": "Anna", "city": "Warsaw", "segment": "premium"},
    2: {"name": "Piotr", "city": "Krakow", "segment": "standard"},
    3: {"name": "Kasia", "city": "Gdansk", "segment": "premium"},
}

@app.get("/customers/{customer_id}")
def get_customer(customer_id: int):
    """Returns customer data by ID."""
    return customers.get(customer_id, {"error": "Customer not found"})

@app.get("/customers/")
def list_customers(segment: str = None):
    """Returns customer list, optionally filtered by segment."""
    if segment:
        return {k: v for k, v in customers.items() if v["segment"] == segment}
    return customers

After starting (uvicorn main:app --reload), the server listens on http://localhost:8000. You can test:

  • GET /customers/1 — Anna’s data
  • GET /customers/?segment=premium — all premium customers
  • GET /docs — automatic API documentation (Swagger UI)

ML Model as a Service

Training a good ML model is just the first step. To make it useful, you need to serve it to end users — most commonly as an API.

from fastapi import FastAPI
import pickle
import numpy as np

app = FastAPI()

# Load trained model
with open("model.pkl", "rb") as f:
    model = pickle.load(f)

@app.get("/predict/")
def predict_price(area: float, bedrooms: int, building_age: int):
    """Real estate price prediction."""
    features = np.array([[area, bedrooms, building_age]])
    price = model.predict(features)[0]
    return {"estimated_price": round(price, 2)}

A request GET /predict/?area=65&bedrooms=3&building_age=10 returns the estimated property price. A million users can simultaneously use the same model.

Key model serving metrics:

  • Latency — response time per request,
  • QPS (Queries Per Second) — number of requests handled per second,
  • Infrastructure cost — how much we pay for servers.

Apache Kafka

Apache Kafka is a distributed platform for stream data processing. It was created at LinkedIn in 2009 and has been developed as an Apache project since 2012.

Kafka is not a regular message queue. The difference is fundamental.

Kafka vs Traditional Queue

In a classical queue system (e.g., RabbitMQ), a message is deleted after being read. One consumer picks it up and it’s gone.

In Kafka, messages are stored on disk for a configured retention period (e.g., 7 days). Multiple consumers can read the same data independently. A consumer that restarts can resume from where it left off.

Kafka Architecture

Kafka consists of several key components:

Broker — a Kafka server. A Kafka cluster is a group of brokers working together. If one goes down, the others take over its responsibilities.

Topic — a logical channel to which producers send messages and from which consumers read them. Think of it as a folder — grouping related events (e.g., topic transactions, topic logs, topic orders).

Partition — a physical division of a topic. Each partition is an ordered, immutable sequence of messages. Partitions enable parallel processing — more partitions means more consumers can read simultaneously.

Offset — a unique message identifier within a partition. A consumer tracks its offset — knowing which messages it has already processed.

Replication — each partition can be replicated across multiple brokers. If one broker fails, data isn’t lost.

Producer

A producer is an application that sends messages to a Kafka topic.

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers="localhost:9092",
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

transaction = {
    "id": "TX5001",
    "amount": 1250.00,
    "store": "Warsaw",
    "type": "purchase"
}

producer.send("transactions", value=transaction)
producer.flush()
print(f"Sent: {transaction}")

How Kafka assigns messages to partitions:

  • Default: round-robin algorithm — successive messages go to successive partitions.
  • With a key: if we provide a message key (e.g., store), Kafka computes its hash and always assigns messages with the same key to the same partition. This guarantees event ordering per key.

Consumer

A consumer reads messages from a topic and processes them.

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    "transactions",
    bootstrap_servers="localhost:9092",
    auto_offset_reset="earliest",
    value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)

print("Listening for transactions...")
for message in consumer:
    t = message.value
    print(f"[offset={message.offset}] {t['id']}: {t['amount']} PLN ({t['store']})")

Consumer Groups

When a single consumer can’t keep up with processing, we add more consumers to a group. Kafka automatically distributes partitions among consumers in the group:

  • Each partition is assigned to exactly one consumer in the group.
  • Ideally: as many consumers as partitions.
  • If a consumer fails, Kafka reassigns its partitions to another consumer (rebalancing).

This mechanism ensures scalability — you add consumers as load grows.

Publish/Subscribe Pattern

Kafka implements the Pub/Sub pattern — a producer publishes messages to a topic, and any number of consumers (from different groups) can independently read them. This is the fundamental difference from classical queues, where a message goes to only one receiver.

Example: a producer sends transactions to the transactions topic. Simultaneously:

  • Group A (anti-fraud system) analyzes them for fraud.
  • Group B (reporting module) generates sales statistics.
  • Group C (recommendation engine) updates customer profiles.

Each group reads the same data independently.

Summary

In this lecture you learned:

  • Microservice architecture and why it’s replacing monoliths.
  • REST APIs with FastAPI — serving data and ML models.
  • Apache Kafka — a distributed streaming platform: brokers, topics, partitions, offsets, producers, consumers, consumer groups.

In the labs, you’ll set up Kafka in Docker and write your own producer and consumer. In the next (final) lecture, we’ll cover machine learning in the context of stream processing.

Business Impact

Microservice architecture built on Kafka gives a company flexibility — new business features can be deployed without stopping the entire system. Serving ML models via API opens the door to API Economy — monetizing algorithms as products for external partners. Containerization (Docker) converts fixed infrastructure costs (Capex) into variable costs (Opex), which is key to data-driven project profitability.