Schema evolution

Overview

Artie supports schema evolution out of the box, which means:

• We automatically create tables in your destination if they don’t exist
• We detect and add new columns automatically
• We can drop deleted columns from the source (optional feature)

How do we pick data types?

• We automatically detect data types from your source schema and values
• For more details, see our typing library documentation

How do I drop deleted columns?

Turn on Drop deleted columns in your destination’s advanced settings.

More controls

We provide additional configuration options to give you more control over schema evolution:

Column exclusion

Use column exclusion to prevent specific columns from being added to your destination. This is particularly useful for:

• Excluding sensitive data columns (e.g., PII, passwords)
• Preventing unnecessary columns from being synced
• Managing storage costs by excluding large or unused columns

To use this feature, specify the columns you want to exclude in your table’s advanced settings.

Column inclusion

Column inclusion allows you to explicitly define which columns should be synced to your destination. This feature:

• Gives you complete control over your destination schema
• Helps maintain a clean, minimal dataset
• Requires manual management of new columns

Automatic schema changes

A workspace admin can turn on Automatically apply supported schema changes in Settings > General, under Table change handling. This runs a separate reconciliation job that compares an eligible source table with its existing destination table and applies supported differences.

Which sources are supported?

Automatic schema changes apply to sources with authoritative column schemas: PostgreSQL, CockroachDB, MySQL, PlanetScale, Microsoft SQL Server, Oracle, and Amazon Keyspaces (Cassandra). The job also requires a destination that supports column metadata and an existing destination table.

When does the job run?

Artie schedules the job daily between 7:00 and 8:00 AM Pacific Time. It also queues reconciliation after an eligible pipeline redeploys, but not after a pipeline’s first deployment. Reconciliation runs asynchronously and does not determine whether a deployment succeeds.

The scheduled job only processes eligible running, unlocked pipelines and applies a two-hour cooldown per pipeline, so repeated checks or redeployments do not necessarily start another reconciliation immediately. Redeploy-triggered reconciliation also does not enqueue when the pipeline’s data plane is locked.

Run a check on demand

Use the Dashboard API when you need to request reconciliation before the next scheduled run:

• POST /company/automatic-schema-changes requests checks for every eligible running pipeline in the authenticated company. An environment-scoped API key only includes pipelines in that environment.
• POST /pipelines/{uuid}/automatic-schema-changes requests a check for one eligible pipeline.

Both requests return 202 Accepted after Artie queues or coalesces the work; they do not wait for schema reconciliation to finish. The company setting must be enabled, and a pipeline request requires a running, unlocked pipeline. Existing source and destination prerequisites, column inclusion and exclusion rules, hard-delete settings, and the two-hour cooldown still apply.

Does it only add columns?

By default, yes: the job adds source columns that are missing from an existing destination table. It does not rename columns or change the type of an existing destination column.

It can also drop a destination column, but only when Drop deleted columns is on in the destination’s advanced settings. Column inclusion and exclusion still determine which source columns Artie can add. The job drops a column once it’s gone from the source table’s schema, without the four-hour check described above.