Artie is a fully managed real-time streaming platform that continuously replicates database changes into warehouses and lakes. We automate the entire data ingestion lifecycle - from capturing changes to merges, schema evolution, backfills, and observability - and scale to billions of change events per day.
This post is for data engineers and platform teams evaluating how to build or buy a real-time streaming pipeline. It assumes basic familiarity with databases and data warehouses.
Key Takeaways
- A real-time data streaming architecture has four layers: ingestion, transport, processing, and serving. A gap in any one of them shows up as latency, data loss, or operational pain.
- Most teams underestimate the maintenance cost of building streaming pipelines in-house - schema evolution, exactly-once delivery, and backpressure handling are where DIY solutions break down.
- Streaming and batch are not mutually exclusive. Many production architectures run both; the key is knowing which workloads justify sub-minute latency.
Why Real-Time Data Streaming Architecture Matters
Everyone says they want real-time data. Fewer teams have actually thought through what that requires architecturally.
The difference between a good real-time data streaming architecture and a bad one isn't theoretical. It shows up in production.
A fintech company scoring transactions for fraud needs results in milliseconds. If the streaming pipeline adds 30 seconds of latency, the fraudulent charge has already been approved. An e-commerce platform running inventory dashboards off nightly batch jobs oversells products because the data is always 8 hours stale. A machine learning team training recommendation models on yesterday's feature data watches their model drift because user behavior shifted hours ago. When Substack moved to real-time replication, they saw a 98% reduction in data latency - their A/B testing framework got faster and the whole company started making decisions quicker.
These aren't edge cases. They're the normal outcomes when architecture decisions don't match latency requirements.
Real-time data streaming is the continuous movement of data from source systems to destinations as changes happen - not on a schedule, not in bulk, but event by event. The architecture around it determines whether you actually achieve that or just end up with a slightly faster batch process.
Getting this right matters more now than it did five years ago. AI applications, operational analytics, and compliance requirements all depend on data that's current - not data that was current six hours ago.
Core Components of a Real-Time Streaming Data Pipeline
A real-time data streaming pipeline has four layers. Each one solves a different problem, and skipping any of them creates gaps that show up as latency, data loss, or operational headaches.
Ingestion is where changes originate. For databases, this usually means Change Data Capture (CDC) - reading row-level changes from the database's transaction log. In Postgres, that means reading from the write-ahead log (WAL) through replication slots. For applications, it's event streams from user activity, IoT sensors, or API calls. When ingestion is unreliable, you get missed changes and growing WAL files that threaten the health of your source database.
A quick way to check if your replication slots are healthy:
SELECT slot_name, active,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;If retained_wal is growing and active is false, your CDC client has fallen behind and WAL is piling up.
Transport is the message bus - Apache Kafka, Redpanda, or Amazon Kinesis. It decouples producers from consumers, buffers events during traffic spikes, and preserves ordering. Without a transport layer, a slow consumer directly impacts your source system.
Processing covers transformations, enrichment, filtering, and aggregation. Stream processors like Apache Flink or ksqlDB handle this work. This is where exactly-once semantics and stateful operations like windowed aggregations matter most.
Serving is where data lands - a warehouse like Snowflake or BigQuery, a data lake on Apache Iceberg or Delta Lake, or an operational system. The serving layer determines how quickly downstream consumers can query fresh data.
Common Real-Time Data Streaming Patterns
Not every real-time data streaming pipeline looks the same. The pattern you choose depends on where data originates, where it needs to go, and what guarantees you need along the way.
Change Data Capture (CDC) streams row-level changes from a database in real time. It's the most common pattern for keeping a warehouse in sync with a transactional database.
CDC reads from the database's transaction log, so it captures every insert, update, and delete without running queries against production tables. The trade-off: the source database has to retain its transaction log until the CDC client catches up. And certain edge cases - like Postgres TOAST columns that omit unchanged large values from the WAL - can silently corrupt data downstream if your pipeline doesn't handle them.
Event sourcing stores every state change as an immutable event in an append-only log. Instead of overwriting the current state, you record the full history - which is useful for audit trails, debugging, and replaying state. The trade-off is replay complexity and storage growth. Reconstructing current state from millions of events gets expensive at scale.
Pub/sub fan-out publishes one event to multiple independent consumers. A single order event might trigger inventory updates, analytics, notifications, and fraud scoring simultaneously. The trade-off: ordering guarantees weaken as consumer count grows, and a slow consumer can create backpressure across the system.
Stream-table join (enrichment) combines a live event stream with a reference table in real time. For example, enriching a transaction event with the customer's risk score before routing it to a fraud model. The trade-off is state management - the reference table has to be kept up to date in memory or a fast lookup store, which introduces memory pressure and consistency windows.
Design Principles for Reliable Real-Time Data Streams
Patterns get you started. Principles keep you running in production.
Exactly-once delivery means every event is applied to the destination exactly one time - no duplicates, no gaps. In practice, most systems achieve this through at-least-once delivery combined with idempotent writes. For example, using MERGE operations with primary keys so that reprocessing an event produces the same result as processing it once.
Idempotency is the foundation that makes retries safe. When a network error forces a retry, the same event processed twice should produce the same outcome. This matters everywhere - from the transport layer acknowledging messages to the serving layer writing rows. Without idempotency, retries create duplicates.
Schema evolution is inevitable. Columns get added, types change, tables get renamed. If your real-time data streaming pipeline can't handle this automatically, someone gets paged at 2am to manually run ALTER TABLE statements in the warehouse. Ask any data engineer how that call goes. Production pipelines need to detect schema changes upstream and propagate them to the destination without manual intervention.
Backpressure handling determines what happens when a consumer can't keep up with the producer. The system has three choices: slow down the producer, buffer events in a durable store like Kafka, or drop data. Dropping data is almost never acceptable. Good architecture uses buffering and consumer-side scaling to absorb spikes without losing events.
Observability is non-negotiable. You need visibility into replication lag, throughput, error rates, and pipeline health. If you can't see that a pipeline is falling behind, you won't know until stale data causes a downstream incident. Dashboards aren't enough - you need alerts that fire before the problem becomes visible to end users.
Real-Time Streaming vs. Batch: When to Use Each
Streaming and batch are not mutually exclusive. Most production data architectures run both. The question isn't which to pick - it's which workloads justify the operational complexity of real-time.
Batch still makes sense for workloads where latency doesn't matter - nightly financial reconciliation, weekly reports, or preparing training data. Streaming makes sense when the cost of stale data exceeds the cost of running a real-time pipeline.
Many teams start with batch and move specific workloads to streaming as latency requirements tighten. That hybrid approach is pragmatic and common.
Building vs. Buying a Real-Time Data Streaming Solution
Building a real-time data streaming pipeline from scratch is doable. Teams do it with Debezium, Kafka, and custom consumers. But the effort that goes into making it production-grade is consistently underestimated.
The initial setup - standing up Kafka, configuring Debezium connectors, writing consumers - takes weeks. But that's the easy part. The hard part is everything that comes after:
- Schema evolution across dozens of tables, where a single column addition shouldn't require a pipeline restart or manual DDL in the warehouse.
- Edge cases like TOAST columns in Postgres, replication slot bloat, and partitioned tables - each one a potential production incident.
- Exactly-once delivery end-to-end, which requires idempotent writes, deduplication logic, and careful failure handling at every stage.
- Backfill without locking production tables or creating data gaps when you add a new table to the pipeline.
- Monitoring and alerting on replication lag, throughput, pipeline health, and source database impact.
These are engineering months, not days. We've talked to teams that spent 9+ months getting a Debezium-Kafka-Snowflake pipeline to production-grade, only to still deal with weekly schema change incidents. And the work is ongoing - every Kafka upgrade, every Debezium patch, every new Postgres version introduces maintenance.
This is where managed platforms earn their place. Artie handles CDC, schema evolution, exactly-once delivery, backfills, and observability out of the box. Pipelines deploy in minutes, run with sub-minute latency, and scale to billions of events per day without a dedicated streaming infrastructure team. If your goal is getting real-time data into your warehouse or lake - not operating Kafka clusters - it's worth evaluating a managed alternative before committing to a multi-quarter build.
FAQ
What is the difference between real-time data streaming and batch processing?
Real-time data streaming processes events continuously as they arrive, delivering sub-minute freshness to downstream systems. Batch processing collects data over a window - hours or a full day - and processes it all at once on a schedule. Streaming suits workloads where stale data is costly, like fraud detection or operational dashboards. Batch suits workloads where simplicity and lower infrastructure cost matter more than immediacy.
What tools are commonly used in a real-time data streaming architecture?
Common tools include Apache Kafka and Redpanda for transport, Apache Flink and ksqlDB for stream processing, and Debezium for open-source CDC. Managed platforms like Artie handle end-to-end real-time replication into warehouses and lakes. AWS Kinesis and Spark Structured Streaming are also widely used in cloud-native architectures. The right tool depends on your latency requirements, team size, and operational appetite.
How does Change Data Capture fit into real-time streaming?
CDC is the ingestion layer of a real-time data streaming pipeline. It captures row-level changes - inserts, updates, deletes - directly from a database's transaction log, like the Postgres WAL. Those changes flow through a transport layer (like Kafka) into processing and then into a destination warehouse or lake. CDC avoids running queries against production tables, which keeps the performance impact on your source database minimal.
What does "exactly-once delivery" mean in streaming data pipelines?
Exactly-once delivery means each event is processed and applied to the destination exactly one time, with no duplicates and no gaps. Most systems achieve this through at-least-once delivery combined with idempotent writes - for example, using MERGE statements keyed on primary keys so that reprocessing a duplicate event doesn't create a duplicate row in the destination.
How much does it cost to run a real-time data streaming pipeline?
Costs depend on data volume, number of sources, destination, and whether you build or buy. A self-managed stack requires at least 3 dedicated engineers with distributed systems expertise (usually more like 5+) - these are people who understand Kafka internals, consumer group rebalancing, and WAL-level replication. At roughly $250K in fully loaded cost per engineer per year, that's $750K-$1.25M/year ($63K-$104K/month) in engineering alone. Add $5-15K/month in Kafka infrastructure, networking, and storage, and you're looking at $68-119K/month before accounting for on-call overhead and incident response. And that's just the ongoing run cost. Building the pipeline in the first place typically takes 6-12 months before it's production-ready - that's 6-12 months where your team doesn't have real-time data and downstream use cases are blocked. Time to value is one of the biggest hidden costs. Managed platforms reduce both the build time (minutes, not quarters) and the ongoing burden by eliminating the engineering and operational overhead entirely. The right comparison isn't just infrastructure cost - it's the full total cost of ownership, including the cost of waiting.
What to Do Next
If you're evaluating real-time data streaming architectures, start by mapping your use cases to latency requirements. Not everything needs sub-minute freshness. But for the workloads that do - fraud detection, operational analytics, AI features, customer-facing data products - the architecture decisions you make now will determine whether real-time actually works or just looks good on a slide.
If you want to skip the 6-12 month build and start streaming in minutes, talk to the Artie team.
