Apache Kafka

Apache Kafka is an open-source platform for publishing, storing and reading streams of records, called events, organized into named topics. Producers write events to a topic and consumers read them, and unlike a traditional queue, events remain on the topic for a configurable retention period so multiple consumers can read the same data independently, and a new consumer can replay history from any earlier point.

Kafka runs as a cluster of brokers, with each topic split into partitions that are distributed and replicated across the cluster for throughput and fault tolerance; ordering is guaranteed within a partition but not across an entire topic. This architecture is what distinguishes Kafka from a simpler message broker: it behaves more like a distributed, append-only commit log than a delivery queue, which is why it underpins event streaming use cases at large scale rather than simple task queuing.

Kafka is the de facto standard companion to event-driven architecture and feeds stream processing engines like Flink and Kafka Streams directly. It matters because it decouples data producers from consumers and can sustain very high write and read throughput. Common pitfalls include treating Kafka as a database for querying, under-provisioning partitions so throughput can't scale later, and letting consumer groups fall behind until retention expires and data is lost before it's read.