Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Exactly-Once Delivery

Exactly-once delivery is a guarantee that each message or change event takes effect in the destination one time: never lost and never applied twice.

What Is Exactly-Once Delivery?

Exactly-once delivery is a guarantee that each message or change event from a source takes effect in the destination one time: never lost and never applied twice, including after crashes, retries, and restarts. Data pipelines achieve it by pairing retried transport with deduplication, idempotent writes, or atomic commits of data and offsets.

A sender that gets no acknowledgment cannot tell whether the message or the acknowledgment was lost, so it retries. Systems advertising exactly-once semantics, such as Kafka and Flink, guarantee an exactly-once effect: duplicates are discarded before they change state.

How Exactly-Once Delivery Works

Exactly-once pipelines retry failed writes, then neutralize the duplicates with three mechanisms, often combined.

Receiver-side deduplication

  • Kafka idempotent producer – the broker tracks a producer ID and per-partition sequence numbers and rejects retried batches; on by default since Kafka 3.0.
  • Snowpipe Streaming offset tokens – Snowflake persists a client-supplied offset token with committed rows; a restarted client calls getLatestCommittedOffsetToken and resumes from there.
  • BigQuery Storage Write API – application-created streams reject a second append at the same offset.

Atomic commit of data and position

The output and the consumed input position commit together, so a crash leaves both or neither.

  • Kafka transactions – a producer with a transactional.id commits writes and consumer offsets together; read_committed consumers skip aborted records. Kafka Streams exposes this as exactly_once_v2.
  • Flink two-phase commit sinks – pre-commit at each checkpoint, commit when it completes.
  • Kafka Connect – KIP-618 (Kafka 3.3) commits source records and source offsets in one transaction.

Idempotent writes

An idempotent write leaves the same table state however many times it runs, such as a MERGE or upsert keyed on the primary key. Idempotency at the sink is the only mechanism that works when source and destination share no transaction.

Exactly-Once vs At-Least-Once vs At-Most-Once

The three delivery guarantees differ in what a system does when an acknowledgment is lost.

GuaranteeOn failureMechanismTrade-off
At-most-onceNever retries; messages can be lostCommit offset before processingLowest overhead; fits only loss-tolerant data
At-least-onceRetries; messages can apply twiceCommit offset after processingDefault for most brokers and CDC tools, including Debezium; needs idempotent consumers
Exactly-onceRetries, then discards duplicatesDedup IDs, transactions, or idempotent sinksExtra coordination; holds only inside the vendor's stated boundary

When to use which: at-least-once delivery plus idempotent writes covers most replication workloads. Transactional exactly-once earns its cost when the destination appends or aggregates.

Exactly-Once Delivery Examples

Exactly-once guarantees matter wherever a duplicate changes a number instead of overwriting a row.

  • CDC to a warehouse – a change data capture pipeline restarting from a Postgres replication slot's confirmed LSN (log sequence number, a write-ahead log position) or a MySQL binlog position re-sends changes. A primary-key MERGE absorbs them; an append-only history table records them twice.
  • Aggregates and ledgers – a replayed event double-counts an order total; a retried ledger insert creates a second debit.
  • External side effects – emails and payment API calls sit outside any pipeline transaction and rely on keys the receiver checks, such as Stripe's Idempotency-Key header.

Challenges of Achieving Exactly-Once Delivery

Exactly-once between two systems is hard because the destination write and the source offset commit are separate operations.

  • The crash window – committing the offset before the write loses data on a crash; writing first replays it. Only an offset stored in the destination's transaction, or an idempotent write, closes the window.
  • Zombie writers – a stalled worker resuming after reassignment can write stale data. Kafka fences these with producer epochs.
  • Scope limits – Kafka's guarantee ends at the Kafka boundary; a consumer writing to Snowflake is at-least-once again.
  • Merge cost – MERGE rewrites target data and costs more than an append. Batching more changes into each merge reduces that overhead.

Exactly-Once Delivery Best Practices

Exactly-once outcomes in replication pipelines come from idempotent writes and offset commits that follow the destination write.

  • Version writes by source log position, so an out-of-order replay cannot overwrite a newer row.
  • Commit the source offset last, after the write succeeds.
  • Validate row counts. Checking rows affected by COPY INTO and MERGE catches writes that succeed but land nothing.
  • Buffer between reader and writer. Separating the log reader from the destination writer keeps destination failures from stalling the source log.

A vendor's exactly-once claim should state which boundary it covers, what happens to rows written between the last offset commit and a crash, and whether append-only tables are deduplicated.

Frequently asked questions

Is exactly-once possible across two separate systems?

Exactly-once across two separate systems is possible only as an outcome: duplicates still cross the network, and the destination discards them. The destination must store the source offset in the same transaction as the data, as Snowpipe Streaming does, or apply every write idempotently.

What is the difference between delivery and processing guarantees?

A delivery guarantee counts how many times a message reaches a receiver. A processing guarantee counts how many times its effect reaches state or output. Kafka Streams and Flink provide exactly-once processing on top of at-least-once delivery. Most vendor exactly-once claims are processing guarantees.

Does exactly-once slow a pipeline down?

Exactly-once mechanisms add overhead. Kafka transactions write commit markers, and read_committed consumers wait on open transactions. Flink two-phase commit sinks expose data only when a checkpoint completes, so latency tracks the checkpoint interval. Warehouse MERGE costs more than append.

How do you test a pipeline for duplicates?

Testing a pipeline for duplicates starts with grouping destination rows by primary key and flagging counts above one. Fault injection completes the test: kill the writer between the destination write and the offset commit, restart it, and rerun both checks.

Watch your data replicate live