Exactly-Once Delivery
Exactly-once delivery is a guarantee that each message or change event takes effect in the destination one time: never lost and never applied twice.
What Is Exactly-Once Delivery?
Exactly-once delivery is a guarantee that each message or change event from a source takes effect in the destination one time: never lost and never applied twice, including after crashes, retries, and restarts. Data pipelines achieve it by pairing retried transport with deduplication, idempotent writes, or atomic commits of data and offsets.
A sender that gets no acknowledgment cannot tell whether the message or the acknowledgment was lost, so it retries. Systems advertising exactly-once semantics, such as Kafka and Flink, guarantee an exactly-once effect: duplicates are discarded before they change state.
How Exactly-Once Delivery Works
Exactly-once pipelines retry failed writes, then neutralize the duplicates with three mechanisms, often combined.
Receiver-side deduplication
- Kafka idempotent producer – the broker tracks a producer ID and per-partition sequence numbers and rejects retried batches; on by default since Kafka 3.0.
- Snowpipe Streaming offset tokens – Snowflake persists a client-supplied offset token with committed rows; a restarted client calls
getLatestCommittedOffsetTokenand resumes from there. - BigQuery Storage Write API – application-created streams reject a second append at the same offset.
Atomic commit of data and position
The output and the consumed input position commit together, so a crash leaves both or neither.
- Kafka transactions – a producer with a
transactional.idcommits writes and consumer offsets together;read_committedconsumers skip aborted records. Kafka Streams exposes this asexactly_once_v2. - Flink two-phase commit sinks – pre-commit at each checkpoint, commit when it completes.
- Kafka Connect – KIP-618 (Kafka 3.3) commits source records and source offsets in one transaction.
Idempotent writes
An idempotent write leaves the same table state however many times it runs, such as a MERGE or upsert keyed on the primary key. Idempotency at the sink is the only mechanism that works when source and destination share no transaction.
Exactly-Once vs At-Least-Once vs At-Most-Once
The three delivery guarantees differ in what a system does when an acknowledgment is lost.
| Guarantee | On failure | Mechanism | Trade-off |
|---|---|---|---|
| At-most-once | Never retries; messages can be lost | Commit offset before processing | Lowest overhead; fits only loss-tolerant data |
| At-least-once | Retries; messages can apply twice | Commit offset after processing | Default for most brokers and CDC tools, including Debezium; needs idempotent consumers |
| Exactly-once | Retries, then discards duplicates | Dedup IDs, transactions, or idempotent sinks | Extra coordination; holds only inside the vendor's stated boundary |
When to use which: at-least-once delivery plus idempotent writes covers most replication workloads. Transactional exactly-once earns its cost when the destination appends or aggregates.
Exactly-Once Delivery Examples
Exactly-once guarantees matter wherever a duplicate changes a number instead of overwriting a row.
- CDC to a warehouse – a change data capture pipeline restarting from a Postgres replication slot's confirmed LSN (log sequence number, a write-ahead log position) or a MySQL binlog position re-sends changes. A primary-key
MERGEabsorbs them; an append-only history table records them twice. - Aggregates and ledgers – a replayed event double-counts an order total; a retried ledger insert creates a second debit.
- External side effects – emails and payment API calls sit outside any pipeline transaction and rely on keys the receiver checks, such as Stripe's
Idempotency-Keyheader.
Challenges of Achieving Exactly-Once Delivery
Exactly-once between two systems is hard because the destination write and the source offset commit are separate operations.
- The crash window – committing the offset before the write loses data on a crash; writing first replays it. Only an offset stored in the destination's transaction, or an idempotent write, closes the window.
- Zombie writers – a stalled worker resuming after reassignment can write stale data. Kafka fences these with producer epochs.
- Scope limits – Kafka's guarantee ends at the Kafka boundary; a consumer writing to Snowflake is at-least-once again.
- Merge cost –
MERGErewrites target data and costs more than an append. Batching more changes into each merge reduces that overhead.
Exactly-Once Delivery Best Practices
Exactly-once outcomes in replication pipelines come from idempotent writes and offset commits that follow the destination write.
- Version writes by source log position, so an out-of-order replay cannot overwrite a newer row.
- Commit the source offset last, after the write succeeds.
- Validate row counts. Checking rows affected by
COPY INTOandMERGEcatches writes that succeed but land nothing. - Buffer between reader and writer. Separating the log reader from the destination writer keeps destination failures from stalling the source log.
A vendor's exactly-once claim should state which boundary it covers, what happens to rows written between the last offset commit and a crash, and whether append-only tables are deduplicated.
Frequently asked questions
Is exactly-once possible across two separate systems?
Exactly-once across two separate systems is possible only as an outcome: duplicates still cross the network, and the destination discards them. The destination must store the source offset in the same transaction as the data, as Snowpipe Streaming does, or apply every write idempotently.
What is the difference between delivery and processing guarantees?
A delivery guarantee counts how many times a message reaches a receiver. A processing guarantee counts how many times its effect reaches state or output. Kafka Streams and Flink provide exactly-once processing on top of at-least-once delivery. Most vendor exactly-once claims are processing guarantees.
Does exactly-once slow a pipeline down?
Exactly-once mechanisms add overhead. Kafka transactions write commit markers, and read_committed consumers wait on open transactions. Flink two-phase commit sinks expose data only when a checkpoint completes, so latency tracks the checkpoint interval. Warehouse MERGE costs more than append.
How do you test a pipeline for duplicates?
Testing a pipeline for duplicates starts with grouping destination rows by primary key and flagging counts above one. Fault injection completes the test: kill the writer between the destination write and the offset commit, restart it, and rerun both checks.