Data Stream Processing: How It Works, Use Cases & Key Benefits
It's 7:15 p.m. on a Friday, and a ramen shop in San Francisco's Mission District is the busiest restaurant on a food delivery app. The kitchen falls 25 minutes behind, but the app keeps quoting 30-minute deliveries, because the job that recalculates prep times runs hourly in Snowflake. By the 8 p.m. run, 45 minutes of customers have seen a wrong ETA.
Data stream processing closes that gap. This post is for data engineers deciding whether a workload needs a stream processor. It assumes you know what a database, a warehouse, and a message queue are.
Artie is a fully managed CDC platform that captures row-level database changes and maintains destination tables in Snowflake, BigQuery, Redshift, Databricks, and other supported destinations. Artie delivers sub-minute latency for supported managed CDC workloads; observed freshness depends on the source, destination, table mode, and workload.
Key Takeaways
- Data stream processing computes continuously over incoming events instead of waiting for a batch schedule. Latency depends on the engine, query, and output: record-at-a-time jobs can respond in milliseconds to seconds, while micro-batches and windows take longer.
- A common stream-processing architecture has four parts: event sources, a durable log like Kafka, a processing engine like Flink, and sinks.
- The hard problems are state, time, and failure: remembering across events, handling late events, and recovering without double-counting.
- It pays off when a decision loses value within minutes. If you only need fresh warehouse tables, CDC replication plus warehouse SQL usually gets you there with fewer moving parts.
What Data Stream Processing Is and How It Differs From Batch Processing
Instead of querying a fixed dataset on a schedule, a stream processor continuously consumes an unbounded sequence of events and can maintain state as it processes them. Depending on the engine and query, processing may be record-at-a-time or in small micro-batches. The prep-time number changes when the orders change, not when the clock says so.
Here's how the three common processing models compare:
| Scheduled batch | Micro-batch streaming | Record-at-a-time streaming | |
|---|---|---|---|
| Input | A bounded dataset (yesterday's orders) | Small slices of a stream, every few seconds | One event at a time, as it arrives |
| When results update | On a schedule | Every trigger interval | Continuously |
| Typical latency | Minutes to hours | Seconds to minutes | Milliseconds to seconds |
| Failure recovery | Rerun the job | Restore checkpointed state, then rerun the slice | Restore saved state and resume |
That last row is the part people underestimate. A failed batch job gets rerun. A failed stream job has to resume mid-flight with its counts intact, without missing or double-counting an order. Most of the engineering in real-time data processing goes into making that work.
How Stream Processing Works: The Architecture Behind It
Event sources. Applications publish events directly: the order service emits order_placed at every checkout. Databases produce them through change data capture (CDC), which reads row-level changes from the database's transaction log. In Postgres, that's the write-ahead log, or WAL.
A durable log. Events land in a log like Apache Kafka, which keeps them in order within each partition for a set retention period. A consumer can fall behind, crash, or reread retained history without losing events, as long as the required records have not expired under the topic's retention policy. The app partitions its order_placed topic by restaurant_id, so every ramen shop order lands in the same partition and is ordered relative to that restaurant. Kafka does not provide a total order across partitions.
The processing engine. Apache Flink splits a job into operators and runs copies in parallel, each owning a slice of the keys. Here's a Flink SQL job that counts orders per restaurant in five-minute windows, a rough measure of how busy each kitchen is:
-- Read order_placed events from Kafka, tolerating events up to 30 seconds late
CREATE TABLE order_events (
order_id BIGINT,
restaurant_id BIGINT,
placed_at TIMESTAMP(3),
WATERMARK FOR placed_at AS placed_at - INTERVAL '30' SECOND
) WITH (
'connector' = 'kafka',
'topic' = 'order_placed',
'properties.bootstrap.servers' = 'kafka:9092',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
-- Count orders per restaurant in tumbling five-minute windows
SELECT
restaurant_id,
window_start,
window_end,
COUNT(*) AS orders_placed
FROM TABLE(
TUMBLE(TABLE order_events, DESCRIPTOR(placed_at), INTERVAL '5' MINUTES)
)
GROUP BY restaurant_id, window_start, window_end;Flink updates each restaurant's count as events arrive, but this query emits one final row per restaurant when each five-minute window closes:
restaurant_id | window_start | window_end | orders_placed
--------------+---------------------+---------------------+--------------
4182 | 2026-10-02 19:10:00 | 2026-10-02 19:15:00 | 31
4182 | 2026-10-02 19:15:00 | 2026-10-02 19:20:00 | 38The WATERMARK line matters more than it looks. Events don't always reach Kafka in order: if the order service retries a failed publish, an order_placed event can land 20 seconds after the order was placed. This watermark treats events up to 30 seconds behind the latest observed event time as on time; once the watermark passes a window's end, Flink can close that window.
One catch: a Kafka partition that stops receiving events holds the watermark back, so windows stop closing. Setting table.exec.source.idle-timeout tells Flink to ignore idle partitions until new events arrive.
The latest-offset setting starts the job from new events only and intentionally skips whatever history Kafka still retains. Use earliest-offset if you want the job to replay that history on its first run.
State and checkpoints. Those running counts are state, and they have to survive a crash. Flink keeps state in memory or on local disk (RocksDB for large state) and periodically snapshots it to storage like S3, along with the Kafka offsets it had reached. If a machine dies, Flink restores the latest completed snapshot and resumes from the recorded source positions. With exactly-once checkpointing, Flink-managed state reflects each input event once; end-to-end exactly-once also requires transactional or idempotent sinks.
Sinks. Results go wherever something acts on them: an alerting service, a Redis cache the ETA service reads, another Kafka topic, or a warehouse table.
Data Stream Processing: Use Cases and Key Benefits
Fraud detection. A new account places eight orders in four minutes, each to a different address with a different card. A stateful job counts orders per account over a sliding ten-minute window and flags it before a driver is dispatched. A nightly report would catch it after the chargebacks arrive.
Live operations. When the ramen shop's five-minute order count passes what the kitchen can handle, the job pushes a longer prep time to the ETA service and alerts the restaurant tablet. Customers see an honest ETA, and fewer cancel.
Dynamic pricing. The app compares open orders to available drivers per delivery zone every minute. When a rainstorm doubles demand in the Mission, the delivery fee rises to pull drivers in while it's still raining.
Fresh features for ML models. The restaurant ranking model uses "orders in the last 15 minutes" as a feature. A stream job keeps it current in Redis, so rankings reflect tonight, not last night.
In each case, the app acts while the information is still useful. For more examples, see our post on real-time analytics architecture and use cases.
The Key Tools in the Stream Processing Ecosystem
Data streaming tools fall into three groups: the log that moves events, the engine that processes them, and the connectors that get data in and out.
Transport. Kafka is the most widely used log. Redpanda speaks the Kafka API with a different internal design, and Amazon Kinesis is the AWS-managed option. Our real-time data streaming architecture post covers this layer in depth.
Processing engines. Each trades power against operational weight:
- Apache Flink is built for sophisticated stateful, event-time processing, with checkpoint-based recovery and exactly-once state guarantees when configured accordingly. It's also the most work to run: a cluster, a state backend to tune, and checkpoint storage to manage. AWS, Confluent, and Ververica sell managed versions.
- Kafka Streams is a Java library that runs inside your own service, so there's no cluster to operate. Its topologies read from and write to Kafka. Your application code can call external systems, but those calls don't get the same recovery guarantees.
- Spark Structured Streaming processes streams as small batches by default. It fits teams already on Spark or Databricks who can live with seconds of latency.
- Apache Beam lets you write a pipeline once and run it on Google Cloud Dataflow, Flink, or Spark.
Streaming databases. Materialize and RisingWave keep SQL views up to date as data arrives, with no streaming job to write. Both speak the Postgres wire protocol.
Getting database changes in and out. Debezium is a widely used open-source option for streaming database changes into Kafka, and you operate it yourself. Artie is a managed option for moving database changes into warehouses. It doesn't run Flink jobs or windowed aggregations; it handles the database-to-warehouse leg, which is the only leg many teams need.
When Stream Processing Is the Right Choice and When It Isn't
Stream processing fits when a decision loses its value within minutes and a system, not a person, acts on the result. It's a poor fit when consumers are dashboards refreshed hourly or daily, when the transformations are SQL your analysts already write, or when nobody can be on call for it. A stream processor costs compute around the clock, a source schema change can break a running job, and reprocessing history is limited by your log's retention.
A middle option covers more workloads than people expect. The app's regional managers want a dashboard of orders, cancellations, and delivery times that's a minute or two old. That doesn't need Flink. It needs the Postgres orders table replicated into Snowflake continuously, with the aggregation as plain SQL in the warehouse.
Fresh Snowflake tables work well for dashboards and operator workflows, where a person reads the data and decides what to do. Automatic actions, like changing a customer's ETA, still need an operational path: a stream processor or service writing to something the app reads, like Redis.
That's the setup Artie runs. It reads changes from the Postgres WAL and keeps supported Snowflake tables current with sub-minute latency. Actual freshness depends on the source, destination, table mode, and workload. Your team doesn't operate Kafka, Debezium, or a stream processor to get there. If most of your streaming plans come down to fresh warehouse tables, try Artie for free or talk to our team before you stand up a cluster.
Before you build anything, answer three questions:
- How fast does the result lose value? If hours, batch is fine. If minutes or seconds, keep going.
- Who acts on the result? A person reading a dashboard can likely use fresh warehouse tables. A system taking automatic action probably needs a stream processor.
- Who runs it at 2 a.m.? If no one can own a Flink cluster, start with a managed streaming engine or a managed streaming database.
FAQ
What is the difference between stream processing and event-driven architecture?
An event-driven architecture is a design where services communicate by publishing and reacting to events: the order service emits order_placed, and the notification service sends a text. Stream processing computes over many events at once, with state and time windows. You can have either one without the other.
Do you need Apache Kafka to do stream processing?
No. Flink can read from Kinesis, Pulsar, files, and databases (through Flink CDC connectors), though Kafka Streams topologies read from and write to Kafka. Most production setups still use a durable log, because it absorbs traffic spikes, lets several consumers read the same events, and allows replay after a bug.
How does stream processing handle late-arriving or out-of-order events?
Engines like Flink process by event time, the timestamp of when something happened, and use watermarks to decide when a time window is complete. Events that arrive after the watermark passes can be dropped, sent to a separate output for review, or used to emit a corrected result. Waiting longer improves completeness but adds latency.
What is stateful vs. stateless stream processing?
Stateless processing handles each event on its own, like filtering out cancelled orders or masking card numbers. Stateful processing remembers information across events, like counting orders per account over ten minutes, joining two streams, or removing duplicates. That state has to be stored, checkpointed, and restored after a failure, which makes stateful jobs harder to operate.
How does stream processing connect to a data warehouse?
There are two common paths. A stream-processing job can write results through a destination-specific sink: for example, a Snowflake sink backed by Snowpipe Streaming, or a connector that writes to an Apache Iceberg table. Or raw database changes are replicated into the warehouse with CDC and transformed there in SQL. Many teams do both: Flink for alerts, CDC for analytics.
