Apache Kafka Real-Time Event Streaming & Distributed Log Processing

Asynchronous event streaming powers modern real-time architectures. Learn how to design robust Kafka producers, partition keys, consumer groups, and dead-letter topics.

Event-Driven Fundamentals

An event is a fact that already happened: OrderPlaced, PaymentCaptured, InventoryAdjusted. Producers append facts to immutable logs; consumers read them at their own pace. This decouples writers from readers and makes replay possible.

Kafka Architecture

Kafka organizes events into topics, split into partitions for parallelism and ordered by offset:

• A topic with 12 partitions allows up to 12 consumers in a group to read concurrently.
• Ordering is guaranteed within a partition, not across a topic.
• Partitions replicate across brokers (default replication factor 3) for fault tolerance.

Partitioning Strategy

The partition key determines ordering and distribution:

// All events for one customer land in the same partition → ordered per customer
ProducerRecord record =
  new ProducerRecord<>("orders", order.customerId(), order);

• Key by the entity whose order you must preserve (customer, order, device).
• Hash a high-cardinality key (e.g., customer ID) to distribute load evenly.
• Avoid a single hot key that skews one partition.

Producer Configuration

acks=all
enable.idempotence=true
linger.ms=5
compression.type=lz4

• acks=all with idempotence prevents data loss without duplicates.
• Linger + batching raises throughput dramatically at negligible latency cost.
• Monitor record-queue-time to detect broker-side backpressure.

Consumer Groups

A consumer group lets you scale reading without losing ordering semantics:

@KafkaListener(topics = "orders", groupId = "order-processor")
public void onOrder(Order order) {
  // exactly-once processing via transactional outbox or idempotent handler
}

• Each partition is consumed by exactly one member of the group.
• Rebalance happens when members join or leave — keep processing idempotent.
• Configure max.poll.interval.ms to avoid losing the lease during slow processing.

Schema Registry

Schemas evolve, consumers don't. Use the Schema Registry with Avro or Protobuf:

• Producer registers the schema and stores it under a versioned ID.
• Consumers fetch the schema by ID and deserialize safely.
• Backward-compatible changes let older consumers keep reading new events.

Dead Letter Topics

Not every event succeeds. A DLQ isolates poison messages:

1. Handler fails after N retries with backoff.
2. Event + original error header land on orders.dlt.
3. A repair job replays the DLQ after fixes.

This keeps the main consumer lagging-free while preserving evidence for investigation.

Spring Cloud Stream Integration

Spring Cloud Stream abstracts binding details:

spring:
  cloud:
    stream:
      bindings:
        order-in-0:
          destination: orders
          group: order-processor

The same code runs against a local broker or a managed Kafka cluster.

Performance Tuning

• Batch size: start at 16KB and measure; batch your records in producers.
• Consumers: tune fetch.max.bytes and max.poll.records to match processing speed.
• Replication: 3 for production; keep min ISR at 2 to avoid losing committed data.
• Backpressure: let Kafka backpressure naturally via large max.poll.records with low poll frequency.

Fault Tolerance

• Replication: data survives broker loss.
• Acks=all + min.insync.replicas=2: no committed data is lost.
• Idempotent consumers: replay-safe processing via deduplication keys.
• Transactional outbox: publish database changes and events atomically.

Conclusion

Kafka rewards those who respect its primitives: partition keys, consumer groups, and schema discipline. Nail those and you get a stream that scales, replays, and survives failures — the backbone of modern real-time systems.