How to choose an event streaming platform

An event streaming platform is a durable, ordered log that producers write to and consumers read from independently, at their own pace. That decoupling is the whole point: a payments service can publish an event once, and an analytics pipeline, a fraud model and a notification service can all read it separately, at different speeds, without the producer knowing or caring who's listening. It's the backbone under real-time data pipelines, CDC tools like Debezium, and any system built around reacting to things as they happen rather than querying a database for the current state.

You don't need this category if your systems talk to each other through direct API calls or scheduled batch jobs and that's working fine — event streaming adds real operational complexity (partitioning, consumer offsets, replay semantics) that only pays off once you have multiple independent consumers of the same event, or a genuine need for sub-second delivery.

Kafka the protocol vs Kafka the thing you operate

Apache Kafka is the de facto standard — not just a product but a protocol and ecosystem that most of this category is defined in relation to. It's entirely free and open source with no vendor and no managed offering of its own: you either self-host and operate a cluster, or you pay someone else to run Kafka-compatible infrastructure for you. That's the fork the rest of the category sits on. Confluent, founded by Kafka's original creators, offers both a self-managed Confluent Platform and a fully managed Confluent Cloud that adds schema registry, 120+ connectors and stream processing on top of the open protocol. Redpanda reimplements the Kafka protocol from scratch in C++ instead of the JVM, staying wire-compatible with existing Kafka clients and tooling while removing ZooKeeper and the JVM from the operational picture, available either self-managed or as managed cloud. If your team already has Kafka expertise and existing Kafka clients, staying protocol-compatible (Confluent or Redpanda) means no rewrite; if you're starting fresh, that compatibility matters less.

Cloud-native managed services that aren't Kafka

Amazon Kinesis, Google Cloud Pub/Sub and Azure Event Hubs are each their own cloud's native streaming service, not Kafka derivatives, though Azure Event Hubs specifically exposes a Kafka-compatible endpoint so existing Kafka producers and consumers can connect with a configuration change rather than a rewrite. Kinesis and Pub/Sub don't offer that compatibility layer — choosing one means committing to that cloud's own API and client libraries, which is a real migration cost if you later need to leave. The upside is zero cluster management and tight native integration with the rest of that cloud's analytics stack (Kinesis with Lambda and Firehose, Pub/Sub with Dataflow and BigQuery, Event Hubs with Stream Analytics and Data Factory).

Multi-tenancy and topic scale

Apache Pulsar is worth knowing about specifically because it solves a problem Kafka doesn't natively: Pulsar separates its serving layer (brokers) from its storage layer (Apache BookKeeper) so each scales independently, giving it native multi-tenancy and support for very large numbers of topics — up to a million per cluster — plus built-in geo-replication. If your requirement is genuinely many isolated tenants or topics sharing one cluster, that's Pulsar's specific reason to exist rather than a general Kafka alternative; for typical single-tenant streaming needs, the difference matters less.

Latency and operational overhead

Redpanda exists on a specific claim: lower latency and simpler operations than JVM-based Kafka, achieved by removing ZooKeeper and running as a single binary. Whether that matters to you depends on how latency-sensitive your workloads are and how much operational pain your current Kafka setup actually causes — for many teams running Kafka at moderate scale, the difference is not the deciding factor. Confluent and the cloud-native services (Kinesis, Pub/Sub, Event Hubs) remove operational overhead a different way: by taking broker management off your plate entirely rather than making the broker itself lighter.

How pricing scales

Every tool here except the two Apache Software Foundation projects is usage-based, but the unit differs. Amazon Kinesis bills per GB written and read (On-Demand) or per shard-hour (Provisioned). Google Cloud Pub/Sub bills per TiB of message throughput delivered plus storage for retained messages, with a modest free allowance. Azure Event Hubs bills per throughput or processing unit-hour depending on tier, with four tiers trading isolation for cost. Confluent meters in its own unit (eCKU) per hour plus separate ingress/egress and storage charges. Redpanda's Serverless tier is pay-as-you-go or annual commit, while its BYOC and self-managed Enterprise Edition are quote-based. Apache Kafka and Apache Pulsar carry no license fee at all — your cost is entirely the infrastructure you run them on, or whatever a managed distribution charges on top.

A shortlist by situation

• Want the open protocol with full control and no vendor: Apache Kafka, self-hosted.
• Want Kafka's ecosystem without operating brokers yourself: Confluent Cloud.
• Want Kafka-compatible streaming with less operational overhead than JVM-based Kafka: Redpanda.
• Need native multi-tenancy or geo-replication out of the box: Apache Pulsar.
• AWS-native and want managed streaming with no Kafka to run: Amazon Kinesis.
• GCP-native, especially feeding Dataflow or BigQuery directly: Google Cloud Pub/Sub.
• Azure-native but want to keep existing Kafka clients working: Azure Event Hubs, via its Kafka-compatible endpoint.

Questions to ask a vendor or in a trial

1. What's the actual measured latency and throughput under your event volume and message size, not a published best case?
2. What happens to consumers during a broker failure or rebalance — how much disruption, and how is exactly-once delivery actually guaranteed, if at all?
3. If you migrate off this platform later, how much of your producer/consumer code has to change — does it depend on a proprietary API or a portable protocol?
4. How is retention priced, and what's the real cost of replaying a large backlog of events?
5. What's included for schema management and validation, and is it a separate paid feature?

Common mistakes

Choosing a cloud-native service (Kinesis, Pub/Sub, Event Hubs) without accounting for the migration cost if you later need to leave that cloud — none of the three non-Kafka-compatible options are drop-in replacements for each other. Underestimating the operational cost of self-hosting open-source Kafka or Pulsar, which is nontrivial at any real scale. Treating "usage-based" as automatically cheap — streaming workloads with high, sustained throughput can cost more on a metered cloud service than on infrastructure you operate yourself. And building on a specific vendor's extensions before confirming you actually need the portability of the open protocol.

For two head-to-head looks at specific pairs, see Apache Kafka vs Confluent and Apache Kafka vs Redpanda. Every tool in this category is listed at /tools/category/event-streaming/.