The Data Almanac
Learn about change data capture, replication, and modern data systems.
Change Data Capture & Replication
28 articles
- ExplainerChange Data Capture in SQL Server: Setup, Limits, StreamingEnable change data capture in SQL Server, read changes by LSN, plan for 3-day retention and log growth, and stream CDC into a warehouse.
- ExplainerReal-Time Data Streaming Architecture: Components, Patterns and ExamplesLearn how a real-time data streaming architecture works: the layers of a streaming data pipeline, CDC patterns, Lambda vs. Kappa, and worked examples.
- How-toHow to Stream Data from Postgres to BigQuery in Real TimeReplace hourly batch loads with CDC: the Postgres setup, how changes merge into BigQuery, the failure modes that break pipelines, and how to validate the data.
- ExplainerData Stream Processing: How It Works, Use Cases & Key BenefitsHow data stream processing works, where stateful event-time computation earns its cost, and when fresh warehouse tables are the simpler choice.
- ExplainerReal-Time Data Ingestion: Architecture, Methods & How It WorksLearn how real-time data ingestion works, when CDC is the right approach, and the operational challenges of running a pipeline at scale.
- Roundup7 Best Tools for Snowflake Replication in Real Time in 2026Compare seven real-time Snowflake replication tools and evaluate CDC latency, schema evolution, source impact, and operational ownership.
- ExplainerPostgreSQL 17 Logical Failover Slots: How CDC Survives a Primary PromotionA bullet-point guide to PostgreSQL 17 logical failover slots, the required configuration, and a tested CDC failover drill.
- ExplainerHow Real-Time Data Synchronization Works and Why It Matters in 2026How real-time data synchronization uses CDC to keep warehouses current with production databases in under a minute.
- ExplainerDatabase Backfill Without Downtime: How Online Backfills WorkHow online CDC backfills keep destination tables fully usable during a resync, using dual-write staging and atomic table swaps.
- ExplainerClickHouse Integration: A Practical Guide for Data Teams in 2026Learn how to integrate ClickHouse Cloud with Postgres, MySQL, and more using CDC. Covers table engines, batching, and real-time pipeline setup.
- How-toAI Data Integration: How to Build Real-Time Pipelines for AI SystemsReal-time AI data integration: how CDC-based pipelines eliminate stale data in AI agents, RAG systems, and fraud detection models.
- ExplainerStream DynamoDB to Redshift with CDCLearn how to stream DynamoDB to Redshift with CDC for real-time analytics, faster reporting, and scalable pipelines for all teams.
- ExplainerMySQL to Redshift Replication: Zero-ETL, DMS, or CDCCompare 4 ways to get MySQL into Redshift: AWS zero-ETL, DMS, S3 export + COPY, and binlog CDC. When zero-ETL is enough, its limits, and setup steps.
- ExplainerPostgreSQL to Snowflake ReplicationStream PostgreSQL to Snowflake in real time with CDC, schema evolution, and low-latency replication using Artie.
- How-toMySQL Replication Best Practices: Security, Performance, and Safe FailoverLearn MySQL replication best practices for security, performance, schema changes, lag reduction, and safe GTID-based failover.
- ExplainerPostgreSQL Replication Slots: Create, Monitor, and DropLearn how to create, monitor, and drop PostgreSQL replication slots. Includes SQL examples for checking WAL size, replica identity, and slot troubleshooting.
- ExplainerUnbounded WAL Growth in Idle AWS RDS Logical ReplicationUnbounded WAL growth on idle AWS RDS: why logical replication slots fill storage and how to prevent it.
- How-toHow to Build Audit Logs Using CDC and SCD Type 2Build audit logs with CDC and SCD Type 2. Learn a step-by-step approach to track every change with full history.
- How-toPostgres Replication Slot 101: How to Capture CDC Without Breaking ProductionLearn how to use Postgres replication slots for CDC safely without risking storage issues or production failures.
- How-toWhy TOAST Columns Break Postgres CDC and How to Fix ItTOAST columns can break Postgres CDC. Learn why jsonb turns NULL and how to fix it without bloating WAL.
- ExplainerPostgres CTID Scanning for Faster BackfillsPostgres CTID scanning enables 10x faster backfills for large tables. Learn how CTID works and boosts performance.
- ExplainerPostgres Write-Ahead LogsPostgres write-ahead logs explained: how WAL works, key settings, and best practices for logical replication.
- How-toMySQL Cascading Changes and Best PracticesLearn MySQL cascading changes and best practices to maintain referential integrity across related tables.
- ExplainerPostgres Table Replica IdentityUnderstand Postgres table replica identity, how it impacts logical replication, and why it matters for CDC.
- ExplainerHistory TableLearn how History Tables capture every CDC change with timestamps and operation types for full audit and replay.
- ExplainerIntroduction to Database ReplicationIntroduction to database replication: ensure data accuracy, low latency, redundancy, and reliable access.
- ExplainerPrevent WAL Growth on AWS Postgres RDSPrevent WAL growth on AWS Postgres RDS. Learn how to avoid replication slot overflow and storage issues.
- ExplainerWhy ELTs Are Here to StayWhy ELTs are here to stay: how modern data stacks shift transformations into the warehouse for flexibility.