Changelog

​

July 29, 2025

​

Self-Serve DynamoDB Backfills

Backfills are a critical step in onboarding new pipelines — especially when you’re working with historical data in DynamoDB. Until now, Artie handled that part for you, kicking off a table export behind the scenes. That worked fine — unless something broke. If your AWS role didn’t have the right permissions or something else went sideways, users were left guessing. Now, that guesswork is gone. Introducing: Self-Serve DynamoDB Backfills When setting up a DynamoDB pipeline, you’ll now see a guided flow in the UI that helps you kick off a backfill. You can export the table directly from your AWS account — or select an existing export if you’ve already started one manually. No more opaque failures. If something’s wrong (like missing permissions), you’ll see it immediately and can fix it yourself — without waiting or wondering. Why this matters:

• DynamoDB backfills are now fully transparent and user-controlled
• Errors (like missing permissions) are surfaced immediately for faster fixes
• Reuse recent exports — no need to start from scratch
• Smoother, more reliable onboarding for new DynamoDB pipelines

Setting up a new DynamoDB pipeline? You’ll see the new backfill flow automatically. Questions? Reach out — we’re here to help.

​

July 22, 2025

​

Parallel Segmented Backfills for Postgres

CTID-based backfills are fast and efficient — especially for large, append-only Postgres tables. They scan directly by physical row location, often outperforming logical queries in stable datasets. But CTIDs come with tradeoffs: they’re slow to initialize for large tables, fragile in dynamic tables where rows update or move, and they can time out in environments with aggressive statement_timeout settings. For teams working with massive, constantly changing Postgres tables, these limitations can stall backfill progress or create reliability risks. Artie now supports schema alignment across environments. Parallel Segmented Backfills offer an alternative path. Instead of relying on CTID, Artie slices tables into logical row segments based on integer primary keys — then parallelizes the work across those chunks. The result: similar performance to CTID backfills, but with stronger guarantees in dynamic environments. We recently helped a customer backfill 8 billion rows in an actively updated Postgres table. CTID-based scans kept timing out and drifting. With Parallel Segmented Backfill, we split the workload across logical row ranges and completed the job — no timeouts, no skipped rows, no guesswork. Why this matters:

• Resilient to updates and vacuuming — row movement doesn’t break backfills
• Offers CTID-level performance with better reliability under load
• Avoids statement_timeout failures in large or busy tables
• Makes backfill behavior predictable and tunable

​

July 17, 2025

​

Improved DDL Support for Cell-Based Architectures

In environments with multiple isolated databases — like production, staging, and dev — schema drift is a persistent risk. Columns added in one cell might not appear in another unless there’s active data flowing through. That means teams looking at the “same” table in Snowflake could be seeing different structures, leading to confusion, bugs, and broken dashboards. This became especially painful for teams whose QA and Dev environments receive little to no traffic. With our previous behavior, tables wouldn’t update unless a row changed — leaving environments out of sync. Artie now supports schema alignment across environments. We’ve introduced a new opt-in job that automatically checks and syncs table schemas across environments — even when there are no row changes. If a new column shows up in production, it’ll get added to dev and staging too, so all environments stay aligned. This feature ensures you get consistent schemas, no matter how much (or little) traffic a database gets. Why this matters

• Guarantees column consistency across environments (prod, staging, dev)
• Eliminates silent schema drift in low-traffic databases
• Supports cell-based and single-tenant architectures out of the box
• Reduces debugging time and improves trust in test environments

​

July 3, 2025

​

External Stage Support for Snowflake

Some teams need more control over where their data goes — and how it gets there. Maybe it’s for compliance. Maybe audit. Or maybe they just don’t want Snowflake touching their data until the very last step. By default, Artie loads delta files into Snowflake using internal staging before merging them into the target table. That worked fine for most workflows — but some teams need an extra layer of control over how data flows through their environment. Now: Artie supports external staging. You can configure Artie to write delta files to a Snowflake external stage — like your own S3 bucket — and we’ll read from there when applying changes to your target table. This gives organizations — like federal agencies using Snowflake Gov Cloud — the ability to use an external stage in their own environment, keeping data fully under their control for things like internal review, validation, or security scanning before deciding to merge into Snowflake. Same fully managed sync. Just with the files landing in your environment first. Why this matters ✅ You keep full visibility into what’s being staged before it’s merged
✅ You can retain delta files for auditing or reprocessing — entirely on your terms
✅ You get tighter control over when data crosses trust boundaries There’s no impact on performance. No extra cost. Just more flexibility, when you need it. Want to turn this on? Let us know — we’ll help you get set up.

​

June 26, 2025

​

Column Control: Include, Exclude, Hash

Not every column needs to make it to your warehouse.

Some fields are sensitive. Some are noisy. Some just don’t belong anywhere near analytics.
Now, you can decide exactly what gets replicated — and what doesn’t — with Artie’s expanded column-level controls. Here’s what’s now possible, per column:
✅ Inclusion — define an allowlist. Only replicate what you explicitly approve; otherwise, ignore
🚫 Exclusion — let most of the table through, but block the columns you don’t want downstream
🔐 Hashing — keep the structure, mask the value. Track fields like user IDs, without exposing data

​

Why this matters:

This isn’t just a cleanup job. It’s control over what leaves prod. Inclusion Sometimes, it’s not about removing sensitive fields — it’s about only sending the ones you trust. Inclusion rules flip the default: instead of replicating everything and hoping exclusions or hashes catch the risky stuff, you define exactly what gets through — and block the rest. What this means:

• Safer by default — no surprises when new columns show up
• Compliance-friendly — ideal for PII and financial data
• Cleaner data — only the fields analytics and ML teams actually need

Exclusion Most of the table is fine. But those one or two fields? No thanks. 
Exclusion rules let you drop what doesn’t belong — without touching your schema. Use it when:

• You’re skipping internal metadata, debug fields, or legacy junk
• You want to trim without breaking things
• You’re migrating slowly and need a guardrail, not a wall

Hashing Some fields need to be trackable — but not readable. 
Hashing keeps them in your pipeline without exposing what’s inside. Reach for hashing when:

• You want to track user behavior across systems without exposing identity
• You’re debugging and need to confirm values match across systems — without logging sensitive data
• You’re sharing a warehouse and want to prevent exposing raw PII to teams that don’t need it
• You only need to know whether or not a value has changed

Want to configure it? 
Column-level rules are set at the source. This guide explains where they belong and why.

​

June 19, 2025

​

CDC for Tables Without Primary Keys

Some tables are weird. No primary key (PK), maybe just a unique index or some composite hack someone added in 2017. Until now, those were off-limits for replication. You can now override PK requirements by specifying a unique index — including composite indexes. Artie will respect the exact column order to ensure optimal performance. Why index-based PK overrides matter: Not every table has a clean PK. Some use unique indexes or composite keys that aren’t formally declared as PKs. Until now, these tables were difficult (or impossible) to replicate. This change addresses one of the most common blockers for CDC at scale. What’s changed:

• PK override: Define row identity with a unique index
• Use composite keys — even if unofficial or unenforced
• Preserve the exact index column order – it affects how changes are captured and impacts query performance during replication (e.g., email, account_id, created_at)

This unlocks flexible replication for legacy systems, denormalized tables, and high-volume sources — without compromising performance. When to use index-based keys:

• Your table lacks a formal PK, but has a unique constraint or index
• You rely on composite keys to identify rows
• You’re dealing with legacy systems or data models that weren’t built with CDC in mind

How to enable key overrides: Reach out to enable key overrides — we’ll help define your index logic and validate it during setup.

​

June 17, 2025

​

Backfill Tuning: Picking the Right Batch Size

You can now control how many rows Artie processes at a time during backfills. The default is now 25,000 rows per chunk (up from 5,000), but you can tune this based on performance vs. load tradeoffs. Why backfill batch size matters: Backfills aren’t one-size-fits-all. Some teams want speed. Others are sensitive to database load and tiptoeing around a production DB at 2am. Until now, everyone got the same batch size of 5,000 rows per chunk. Now you can tune backfills to match your style:

• The default is 25,000 rows — we benchmarked a bunch of sizes. 25,000 rows won out. So that’s our new default
• You have control — adjust the batch size to fit your environment

How to tune batch size based on your workload:

• Speed up backfills: Larger chunks = fewer queries, can improve throughput, but overly large chunks can backfire. It’s about finding balance.
• Reduce DB load: Smaller chunks = faster queries, lower impact on source

Need help tuning batch size? If you’re unsure what batch size is right for your workload, reach out — we’ll help you tune it.

​

Read Once and Write to Multiple Destinations

You can now sync data from a single database to multiple destinations — all from the same connector. Why this matters:

• Reduce load on production databases by avoiding duplicate reads and minimizing replication slot overhead
• Ability to fan out to multiple tools — e.g., write to both Snowflake and Redshift
• Ability to support diverse use cases in parallel — analytics, ML, real-time alerting

This feature is designed for organizations that:

• Operate across multiple data platforms 
• Serve many internal teams with different tools
• Need to scale data infrastructure without increasing operational burden

If you’re planning a multi-destination architecture, we’d be happy to help — just reach out.

​

June 2, 2025

​

Iceberg Support Using S3 Tables

This launch adds something big: support for Apache Iceberg using S3 Tables. Artie customers can now:

• Stream high-volume datasets into Iceberg-backed tables stored on S3
• Use S3 Tables’ fully managed catalog, compaction, and snapshot management
• Query efficiently with Spark SQL (via EMR + Apache Livy) without wrestling with cluster glue
• Get up to 3x faster query performance thanks to automatic background compaction

Why is Iceberg a big deal? Because it solves what’s frustrating and limiting about traditional S3-based data lakes. Hive tables are rigid and brittle, with no snapshotting or time travel. Delta Lake is powerful but tied to the Databricks ecosystem. Plain S3 file storage? No metadata layer, no transactions, no query optimizations. Instead, Iceberg gives you a fully open, cloud-native table format with smooth schema evolution, hidden partitions, snapshot isolation, and time-travel queries – all with broad engine support (Spark, Trino, Flink, Presto, Hive). We’re excited about this because it means Artie customers can confidently move massive data volumes without needing to hand-build the plumbing – Iceberg and S3 Tables handle schema changes, partitioning, compaction, and snapshot management behind the scenes, so the system scales cleanly without brittle, custom workflows. 📚 Want to set up Iceberg-backed pipelines? Docs to get started: https://artie.com/docs/destinations/iceberg/s3tables

​

May 14, 2025

​

S3 Iceberg destination (Beta)

S3 Iceberg is now available in beta! This new destination uses AWS’s recently released S3 Tables support, allowing you to replicate directly into Apache Iceberg tables backed by S3. It’s a big unlock for teams building modern lakehouse architectures on open standards.

​

Column Inclusion Rules

You can now define an explicit allowlist of columns to replicate - ideal for PII or other sensitive data. This expands our column-level controls alongside column exclusion and hashing. Only the fields you specify get replicated. Everything else stays out.

​

Autopilot for New Tables

Stop manually hunting for new tables in your source DB. Autopilot finds and syncs them for you - zero config required. Turn it on via: Deployment → Destination Settings → Advanced Settings → “Auto-replicate new tables”

​

Data Quality: Rows Affected Checks

To further enhance the data integrity built into our pipeline, we’ve added another guardrail: verifying the number of rows affected during each database operation. For example, during merge steps (such as in Snowflake), we confirm ROWS_LOADED from copy commands and validate the totals for inserted, updated, or deleted rows. This approach reinforces the robustness of our data replication process and it’s another way we catch issues early and ensure replication integrity.

​

Read Once, Write Many

We recently launched the ability to read-once and write to multiple destinations. This means you no longer need multiple replication slots on your source database. For example, by reading data just once from your Postgres instance and simultaneously replicating it to Snowflake and Redshift, you reduce database overhead and simplify replication architecture.‍

​

Multi-Data Plane Support

Artie now supports hosting pipelines across multiple data planes, whether you’re on our cloud or using your own (BYOC) infrastructure. For example, run one pipeline from Postgres to Snowflake in AWS US-East-1 and another from MySQL to Snowflake in AWS US-West-2.

​

Oracle Fan-in

With our Oracle Fan-in feature, you can now easily replicate data from thousands of Oracle sources - without painful manual setups or infrastructure overload. Fan-in reduces your Kafka topic sprawl, lowers infrastructure costs, and simplifies real-world, complex data replication.