Event-Driven Architecture with Kafka: Designing Events That Scale

Event-driven architecture with Kafka fails or succeeds on one decision: what goes inside the event. This guide walks through three iterations of event schema design, the consumer configuration that actually holds under load, and why idempotency — not exactly-once semantics — is what makes the system reliable.

The migration began with an outage. A payment provider's latency spiked, the synchronous order pipeline backed up behind it, and the whole system was down for 47 minutes at a load of 12,000 orders an hour. Nothing had crashed. Every service was waiting on the one in front of it.

Designing the Event Schema

Three iterations, each teaching something the previous one hid.

Iteration 1: Fat Events

The instinct is to put everything in the event — the full order object, the full customer record — so consumers never need to ask anyone for anything. It works immediately and rots quickly. Every consumer becomes coupled to the producer's internal schema, so a field rename upstream breaks services that never knowingly depended on it.

Iteration 2: Thin Events

The overcorrection is to publish IDs only and let consumers fetch what they need. This restores decoupling and reintroduces the original problem: one event now triggers a burst of synchronous lookups. The cascading failure that motivated the migration comes back wearing a different hat.

Iteration 3: Right-Sized Events

The rule that worked: carry the immutable facts you own, plus stable reference IDs for everything else. An order-placed event carries the order total and line items — facts that will never change and that the order service owns — and a customer ID rather than a customer object. Consumers get what they need to act without a lookup, and nothing couples them to another service's evolving schema.

Consumer Configuration That Holds

Defaults will not survive production. Three settings do most of the work:

Idempotency: The Pattern Nobody Gets Right First

Every tutorial says "make your consumers idempotent" and stops there. Concretely, it means two mechanisms working together.

An idempotency-key table records every processed event ID; the consumer checks and inserts inside the same database transaction as the business write, so a redelivery finds the key already present and skips. A transactional outbox handles the other direction: events to publish are written to a table in the same transaction as the state change, then relayed to Kafka by a separate process. Without it, a service can commit a database change and crash before publishing, leaving the rest of the system permanently unaware.

Exactly-Once Is Narrower Than It Sounds

Kafka's exactly-once semantics — enable.idempotence: true, isolation-level: read_committed, a transactional ID prefix — are real, and they apply only within Kafka. The moment a consumer writes to a database, calls an API, or sends an email, that guarantee ends.

For anything crossing that boundary, idempotent consumers with transactional writes are the actual solution. Exactly-once configuration is worth enabling; it is not a substitute for designing consumers that can safely see the same message twice.

Monitoring and the Observability Gap

Kafka's own metrics tell you about Kafka, not about whether the business process completed. Two additions closed the gap: propagating a correlation ID through every event so a single order can be traced across services with OpenTelemetry, and a small service exposing consumer lag with alerting once lag exceeds 120 seconds.

The Results

What We Would Do Differently

Consumer lag is the metric that predicts incidents. It rises before anything fails, which makes it the one worth paging on.

After migration: 4x throughput, p99 latency from 12 seconds to under 800 milliseconds, and no further cascading failures. A slow downstream provider now produces a growing queue and an alert instead of an outage.

The whole migration took six months. Most of that was not Kafka; it was learning what belonged inside an event.

We design and build event-driven backends and the operational tooling around them. Explore our engineering services or get in touch.