> Markdown version of https://www.artie.com/glossary/exactly-once-delivery. Site index for AI agents: https://www.artie.com/llms.txt

[Back to glossary](https://www.artie.com/glossary)

# Exactly-Once Delivery

Exactly-once delivery is a guarantee that each message or change event takes effect in the destination one time: never lost and never applied twice.

[Share on LinkedIn (opens in a new tab)](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.artie.com%2Fglossary%2Fexactly-once-delivery) [Share on X (opens in a new tab)](https://x.com/intent/post?text=Exactly-Once%20Delivery\&url=https%3A%2F%2Fwww.artie.com%2Fglossary%2Fexactly-once-delivery)

## What Is Exactly-Once Delivery?

Exactly-once delivery is a guarantee that each message or change event from a source takes effect in the destination one time: never lost and never applied twice, including after crashes, retries, and restarts. Data pipelines achieve it by pairing retried transport with deduplication, idempotent writes, or atomic commits of data and offsets.

A sender that gets no acknowledgment cannot tell whether the message or the acknowledgment was lost, so it retries. Systems advertising exactly-once semantics, such as Kafka and Flink, guarantee an exactly-once effect: duplicates are discarded before they change state.

## How Exactly-Once Delivery Works

Exactly-once pipelines retry failed writes, then neutralize the duplicates with three mechanisms, often combined.

### Receiver-side deduplication

* Kafka idempotent producer – the broker tracks a producer ID and per-partition sequence numbers and rejects retried batches; on by default since Kafka 3.0.
* Snowpipe Streaming offset tokens – Snowflake persists a client-supplied offset token with committed rows; a restarted client calls `getLatestCommittedOffsetToken` and resumes from there.
* BigQuery Storage Write API – application-created streams reject a second append at the same offset.

### Atomic commit of data and position

The output and the consumed input position commit together, so a crash leaves both or neither.

* Kafka transactions – a producer with a `transactional.id` commits writes and consumer offsets together; `read_committed` consumers skip aborted records. Kafka Streams exposes this as `exactly_once_v2`.
* Flink two-phase commit sinks – pre-commit at each checkpoint, commit when it completes.
* Kafka Connect – KIP-618 (Kafka 3.3) commits source records and source offsets in one transaction.

### Idempotent writes

An idempotent write leaves the same table state however many times it runs, such as a `MERGE` or upsert keyed on the primary key. Idempotency at the sink is the only mechanism that works when source and destination share no transaction.

## Exactly-Once vs At-Least-Once vs At-Most-Once

The three delivery guarantees differ in what a system does when an acknowledgment is lost.

| Guarantee     | On failure                          | Mechanism                                    | Trade-off                                                                              |
| ------------- | ----------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------- |
| At-most-once  | Never retries; messages can be lost | Commit offset before processing              | Lowest overhead; fits only loss-tolerant data                                          |
| At-least-once | Retries; messages can apply twice   | Commit offset after processing               | Default for most brokers and CDC tools, including Debezium; needs idempotent consumers |
| Exactly-once  | Retries, then discards duplicates   | Dedup IDs, transactions, or idempotent sinks | Extra coordination; holds only inside the vendor's stated boundary                     |

When to use which: at-least-once delivery plus idempotent writes covers most replication workloads. Transactional exactly-once earns its cost when the destination appends or aggregates.

## Exactly-Once Delivery Examples

Exactly-once guarantees matter wherever a duplicate changes a number instead of overwriting a row.

* CDC to a warehouse – a [change data capture](https://www.artie.com/glossary/change-data-capture) pipeline restarting from a Postgres [replication slot](https://www.artie.com/glossary/replication-slot)'s confirmed LSN (log sequence number, a write-ahead log position) or a MySQL binlog position re-sends changes. A primary-key `MERGE` absorbs them; an append-only history table records them twice.
* Aggregates and ledgers – a replayed event double-counts an order total; a retried ledger insert creates a second debit.
* External side effects – emails and payment API calls sit outside any pipeline transaction and rely on keys the receiver checks, such as Stripe's `Idempotency-Key` header.

## Challenges of Achieving Exactly-Once Delivery

Exactly-once between two systems is hard because the destination write and the source offset commit are separate operations.

* The crash window – committing the offset before the write loses data on a crash; writing first replays it. Only an offset stored in the destination's transaction, or an idempotent write, closes the window.
* Zombie writers – a stalled worker resuming after reassignment can write stale data. Kafka fences these with producer epochs.
* Scope limits – Kafka's guarantee ends at the Kafka boundary; a consumer writing to Snowflake is at-least-once again.
* Merge cost – `MERGE` rewrites target data and costs more than an append. [Batching more changes into each merge](https://www.artie.com/blogs/multi-step-merge) reduces that overhead.

## Exactly-Once Delivery Best Practices

Exactly-once outcomes in replication pipelines come from idempotent writes and offset commits that follow the destination write.

* Version writes by source log position, so an out-of-order replay cannot overwrite a newer row.
* Commit the source offset last, after the write succeeds.
* Validate row counts. [Checking rows affected](https://www.artie.com/blogs/improve-data-quality-with-in-flight-data-pipeline-validation) by `COPY INTO` and `MERGE` catches writes that succeed but land nothing.
* Buffer between reader and writer. [Separating the log reader from the destination writer](https://www.artie.com/blogs/not-all-cdc-pipelines-are-created-equal) keeps destination failures from stalling the source log.

A vendor's exactly-once claim should state which boundary it covers, what happens to rows written between the last offset commit and a crash, and whether append-only tables are deduplicated.

## Frequently asked questions

Is exactly-once possible across two separate systems?

Exactly-once across two separate systems is possible only as an outcome: duplicates still cross the network, and the destination discards them. The destination must store the source offset in the same transaction as the data, as Snowpipe Streaming does, or apply every write idempotently.

What is the difference between delivery and processing guarantees?

A delivery guarantee counts how many times a message reaches a receiver. A processing guarantee counts how many times its effect reaches state or output. Kafka Streams and Flink provide exactly-once processing on top of at-least-once delivery. Most vendor exactly-once claims are processing guarantees.

Does exactly-once slow a pipeline down?

Exactly-once mechanisms add overhead. Kafka transactions write commit markers, and `read_committed` consumers wait on open transactions. Flink two-phase commit sinks expose data only when a checkpoint completes, so latency tracks the checkpoint interval. Warehouse `MERGE` costs more than append.

How do you test a pipeline for duplicates?

Testing a pipeline for duplicates starts with grouping destination rows by primary key and flagging counts above one. Fault injection completes the test: kill the writer between the destination write and the offset commit, restart it, and rerun both checks.

## Related resources

* [Glossary term](https://www.artie.com/glossary/change-data-capture)
  ### [Change Data Capture](https://www.artie.com/glossary/change-data-capture)
* [Glossary term](https://www.artie.com/glossary/replication-slot)
  ### [Replication Slot](https://www.artie.com/glossary/replication-slot)
* [Blog post](https://www.artie.com/blogs/multi-step-merge)
  ### [Multi-step merge](https://www.artie.com/blogs/multi-step-merge)
* [Blog post](https://www.artie.com/blogs/improve-data-quality-with-in-flight-data-pipeline-validation)
  ### [In-flight data pipeline validation](https://www.artie.com/blogs/improve-data-quality-with-in-flight-data-pipeline-validation)
* [Blog post](https://www.artie.com/blogs/not-all-cdc-pipelines-are-created-equal)
  ### [Not all CDC pipelines are created equal](https://www.artie.com/blogs/not-all-cdc-pipelines-are-created-equal)

## Watch your data replicate live

[Start for free](https://app.artie.com/signup) [Book a demo](https://www.artie.com/contact)
