Event streaming

Event streaming is the practice of capturing each discrete thing that happens, a click, an order, a sensor reading, as an immutable event and publishing it to a continuously available, ordered log that other systems can read from, either in real time or later. Platforms like Apache Kafka, Amazon Kinesis and Google Pub/Sub are built specifically for this pattern.

What distinguishes event streaming from a generic message broker is durability and replay: events are retained on the log for a configurable period, sometimes indefinitely, so a new consumer can start reading from the beginning of the stream or from any earlier point, not just messages sent after it connected. Multiple independent consumers can read the same stream at their own pace without interfering with each other, which a traditional queue that deletes a message once it is consumed cannot easily support.

Event streaming underpins event-driven architectures and feeds stream-processing engines that compute results as events arrive. It matters because it decouples the system producing data from every system that consumes it, letting new consumers be added later without changing the producer. The main pitfalls are treating the stream as a database substitute when strict transactional guarantees are needed, and underestimating how ordering and exactly-once processing guarantees vary between platforms and configurations.