Streaming Analytics

What is Streaming Analytics?

Streaming analytics is the continuous computation of metrics, aggregates and alerts over event data as each event arrives, rather than after the data is stored and queried on a schedule. A streaming query runs indefinitely and updates its result incrementally. Teams use it for fraud scoring, operational monitoring and live dashboards, where an hours-old answer has lost its value.

How Streaming Analytics Works

A streaming query runs continuously against an unbounded input, such as a Kafka topic. It reads each event, updates state and emits a new result when that state changes.

Ingest from a durable log

Database events usually arrive through change data capture, which reads committed row changes from the transaction log of an OLTP database: the Postgres WAL (write-ahead log) or the MySQL binlog, and publishes each change as an event. The log orders events within a partition and lets a consumer re-read from a stored offset after a failure.

Keep state between events

Aggregations need state, such as running counts or the latest value per key. Apache Flink keeps state in a backend such as RocksDB and snapshots it in periodic checkpoints; Kafka Streams backs local stores with changelog topics. On failure, the job restores the last checkpoint and rewinds the source offset.

Group events into windows

• Tumbling windows are fixed and non-overlapping, such as orders per minute.
• Hopping windows have a fixed size and advance by a smaller step.
• Session windows close per key after a gap of inactivity.

Emit results

Engines emit a final value when a window closes or an update whenever a result changes; update streams need a sink that upserts by key. Streaming SQL expresses all of this declaratively, as in this per-minute count in Flink SQL:

SELECT window_start, merchant_id, COUNT(*) AS orders
FROM TABLE(
  TUMBLE(TABLE orders, DESCRIPTOR(order_ts), INTERVAL '1' MINUTE)
)
GROUP BY window_start, window_end, merchant_id;

Streaming Analytics vs Batch Analytics

Batch analytics runs a query over a bounded dataset on a schedule; the streaming model runs one over an unbounded dataset without stopping.

When to use which: streaming fits results whose value decays within minutes; batch fits everything else, especially full-history joins. Snowpipe Streaming sits between them: it lands rows in Snowflake with low latency, but queries against those rows still run on demand.

Streaming Analytics Examples

• Transactions per card in a 10-minute hopping window, feeding a fraud rule.
• p99 API latency per endpoint per minute, computed with an approximate sketch such as t-digest, since an exact percentile keeps every value in state.
• Units on hand per SKU, maintained from CDC inserts, updates and deletes on the stock table.
• Session length per user, using session windows that close after 30 idle minutes.
• Payments enriched with the merchant's current risk tier through a temporal join against a CDC-fed table.

Streaming Analytics Use Cases

Artie has covered the data feedback loop and which use cases need real-time data, and walked through real-time analytics use cases for a delivery platform.

• Payments: fraud scoring at authorization, exposure monitoring.
• E-commerce and marketplaces: inventory availability, dynamic pricing, delivery ETAs.
• Observability and security: error-rate alerting, intrusion detection rules.
• IoT and logistics: sensor thresholds, fleet tracking.
• Ad tech and gaming: budget pacing, live leaderboards, usage-based billing meters.

Streaming Analytics Challenges

Out-of-order events

Event time, when an event happened, and processing time, when the engine reads it, diverge because of network delay, buffering and retries. Windowing on processing time avoids the problem but changes results on every replay.

Late data and watermarks

A watermark is the engine's estimate that no events older than a given timestamp are still in flight. A window fires once the watermark passes its end; later events are late. Flink SQL window aggregations drop them, Kafka Streams accepts them until the grace period ends, and Spark Structured Streaming may drop data older than the watermark threshold. A tighter bound lowers latency and drops more data.

Operational complexity

• State size grows with key cardinality and window length, slowing checkpoints and recovery.
• Upstream schema changes can break deserialization in a running job.
• Reprocessing a logic bug requires log retention covering the affected period.
• Exactly-once output depends on the sink: transactional or idempotent sinks keep it, plain appends do not.