The Data Almanac
Learn about change data capture, replication, and modern data systems.
All topics
44 articles
- ExplainerReal-Time Data Streaming Architecture: Components, Patterns and ExamplesLearn how a real-time data streaming architecture works: the layers of a streaming data pipeline, CDC patterns, Lambda vs. Kappa, and worked examples.
- How-toHow to Stream Data from Postgres to BigQuery in Real TimeReplace hourly batch loads with CDC: the Postgres setup, how changes merge into BigQuery, the failure modes that break pipelines, and how to validate the data.
- ExplainerData Stream Processing: How It Works, Use Cases & Key BenefitsHow data stream processing works, where stateful event-time computation earns its cost, and when fresh warehouse tables are the simpler choice.
- ExplainerReal-Time Data Ingestion: Architecture, Methods & How It WorksLearn how real-time data ingestion works, when CDC is the right approach, and the operational challenges of running a pipeline at scale.
- Roundup7 Best Tools for Snowflake Replication in Real Time in 2026Compare seven real-time Snowflake replication tools and evaluate CDC latency, schema evolution, source impact, and operational ownership.
- ExplainerPostgreSQL 17 Logical Failover Slots: How CDC Survives a Primary PromotionA bullet-point guide to PostgreSQL 17 logical failover slots, the required configuration, and a tested CDC failover drill.
- ExplainerHow Real-Time Data Synchronization Works and Why It Matters in 2026How real-time data synchronization uses CDC to keep warehouses current with production databases in under a minute.
- ExplainerDatabase Backfill Without Downtime: How Online Backfills WorkHow online CDC backfills keep destination tables fully usable during a resync, using dual-write staging and atomic table swaps.
- ExplainerClickHouse Integration: A Practical Guide for Data Teams in 2026Learn how to integrate ClickHouse Cloud with Postgres, MySQL, and more using CDC. Covers table engines, batching, and real-time pipeline setup.
- How-toAI Data Integration: How to Build Real-Time Pipelines for AI SystemsReal-time AI data integration: how CDC-based pipelines eliminate stale data in AI agents, RAG systems, and fraud detection models.
- ExplainerStream DynamoDB to Redshift with CDCLearn how to stream DynamoDB to Redshift with CDC for real-time analytics, faster reporting, and scalable pipelines for all teams.
- How-toHow to Replicate MySQL to Redshift in Real TimeLearn how MySQL CDC streams changes to Redshift in real time, with setup steps, MERGE patterns, and production tips.
- ExplainerPostgreSQL to Snowflake ReplicationStream PostgreSQL to Snowflake in real time with CDC, schema evolution, and low-latency replication using Artie.
- How-toMySQL Replication Best Practices: Security, Performance, and Safe FailoverLearn MySQL replication best practices for security, performance, schema changes, lag reduction, and safe GTID-based failover.
- ExplainerPostgreSQL Replication Slots: Create, Monitor, and DropLearn how to create, monitor, and drop PostgreSQL replication slots. Includes SQL examples for checking WAL size, replica identity, and slot troubleshooting.
- ExplainerUnbounded WAL Growth in Idle AWS RDS Logical ReplicationUnbounded WAL growth on idle AWS RDS: why logical replication slots fill storage and how to prevent it.
- How-toHow to Build Audit Logs Using CDC and SCD Type 2Build audit logs with CDC and SCD Type 2. Learn a step-by-step approach to track every change with full history.
- How-toPostgres Replication Slot 101: How to Capture CDC Without Breaking ProductionLearn how to use Postgres replication slots for CDC safely without risking storage issues or production failures.
- How-toWhy TOAST Columns Break Postgres CDC and How to Fix ItTOAST columns can break Postgres CDC. Learn why jsonb turns NULL and how to fix it without bloating WAL.
- ExplainerPostgres CTID Scanning for Faster BackfillsPostgres CTID scanning enables 10x faster backfills for large tables. Learn how CTID works and boosts performance.
- ExplainerPostgres Write-Ahead LogsPostgres write-ahead logs explained: how WAL works, key settings, and best practices for logical replication.
- How-toMySQL Cascading Changes and Best PracticesLearn MySQL cascading changes and best practices to maintain referential integrity across related tables.
- ExplainerPostgres Table Replica IdentityUnderstand Postgres table replica identity, how it impacts logical replication, and why it matters for CDC.
- ExplainerHistory TableLearn how History Tables capture every CDC change with timestamps and operation types for full audit and replay.
- ExplainerIntroduction to Database ReplicationIntroduction to database replication: ensure data accuracy, low latency, redundancy, and reliable access.
- ExplainerPrevent WAL Growth on AWS Postgres RDSPrevent WAL growth on AWS Postgres RDS. Learn how to avoid replication slot overflow and storage issues.
- ExplainerWhy ELTs Are Here to StayWhy ELTs are here to stay: how modern data stacks shift transformations into the warehouse for flexibility.
- ExplainerWhat Is a Real-Time Data Warehouse? Architecture, Benefits & Use CasesLearn how real-time data warehouses use CDC and streaming ingestion to keep analytics current, plus architecture, trade-offs, and practical use cases.
- ExplainerSnowflake Schema Evolution: Managing Schema Changes in ProductionHow Snowflake schema evolution handles new columns, where native support stops, and how real-time CDC pipelines keep source and destination schemas aligned.
- ExplainerReal-Time Analytics: Architecture, Use Cases & What Makes It HardLearn real-time analytics architecture, important use cases, and why building low-latency, scalable data systems is so hard today.
- ExplainerHow Apache Iceberg Works: Catalogs, Manifests, and Snapshots ExplainedHow Apache Iceberg works: explore table formats, metadata layers, snapshots, and how Iceberg enables ACID data lakes and faster queries.
- ComparisonThe Ultimate Guide to Snowpipe vs Snowpipe StreamingSnowpipe vs Snowpipe Streaming: key differences, trade-offs, and how to choose for real-time Snowflake ingestion.
- ComparisonHow to choose a data warehouseHow to choose a data warehouse. Compare Snowflake, Redshift, BigQuery, and ClickHouse by key factors.
- How-toBest Practices on Running Redshift at ScaleBest practices for running Redshift at scale. Improve performance and reduce costs with five high-impact tuning strategies.
- ExplainerSnowflake Eco ModeSnowflake Eco Mode reduces compute costs by syncing more during peak hours and scaling back off-peak.
- ExplainerWhat Are the Differences Between OLTP and OLAP Databases?OLTP vs OLAP explained: key differences, workloads, and real-world examples for transactional and analytical databases.
- Roundup8 Best Fivetran Alternatives in 2026: Pricing, CDC and Latency ComparedCompare the 8 best Fivetran alternatives in 2026 for data replication by pricing model, CDC support and latency: Artie, Airbyte, AWS DMS, Datastream, Debezium.
- Roundup8 Best Matillion Alternatives in 2026
- RoundupTop 10 Database Replication Solutions in 2026
- RoundupBest 5 Real-time Data Pipeline Platforms for AI ApplicationsCompare the 5 best real-time data pipeline platforms for AI applications with features for streaming, inference, and scalable ops.
- RoundupBest CDC Tools for Cloud Data Warehouses in 2026: 8 ComparedCompare the best CDC tools for cloud data warehouses in 2026: Artie, Debezium, Fivetran, Airbyte, AWS DMS, and more on latency, sources, and pricing.
- RoundupBest 10 Data Replication Tools for 2026Compare the best data replication tools for real-time CDC in 2026, with features and use cases to guide selection.
- RoundupBest 5 Debezium Alternatives to Consider in 2026Compare the best Debezium alternatives in 2026, including managed CDC, Kafka-based options, and low-latency pipeline tools.
- Roundup4 DMS Alternatives in 2026Compare 4 AWS DMS alternatives for 2026, including CDC and streaming tools for low-latency, reliable data replication.