> ## Documentation Index
> Fetch the complete documentation index at: https://artie.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema evolution

> Learn how Artie automatically evolves schemas to match your data.

## Overview

Artie supports schema evolution out of the box, which means:

* We automatically **create** tables in your destination if they don't exist
* We detect and **add new columns** automatically
* We can drop deleted columns from the source (optional feature)

### How do we know the right data types? 🔍

* We automatically detect data types from your source schema and values
* For more details, see our [typing library documentation](/docs/guides/artie/arties-typing-library#data-type-handling-%F0%9F%94%8D)

<Accordion title="How do I opt-in to dropping deleted columns? ⚙️">
  You can enable this feature by setting `Enable hard deletes for columns` to `true` in your destination's advanced settings.

  <img src="https://mintcdn.com/artie/cR74rDu7gj_LCvTI/assets/hard_delete_columns.png?fit=max&auto=format&n=cR74rDu7gj_LCvTI&q=85&s=6e4e53cd6cdb9c0af466fe07772de787" alt="Enable hard deletes for columns" width="952" height="463" data-path="assets/hard_delete_columns.png" />

  <Note>
    For safety reasons, columns are not automatically dropped from destination tables since this operation is nearly irreversible. While some destinations support undos, we implement a strict safety check process:

    1. When a column is deleted from the source, the first row that omits this column marks it for potential removal
    2. We then monitor CDC events for 4 hours to ensure the column is consistently omitted
    3. Only after this verification period will we proceed with dropping the column

    This safety mechanism helps prevent accidental data loss and ensures the column is truly no longer needed.
  </Note>
</Accordion>

### Advanced Features 🛠️

We provide additional configuration options to give you more control over schema evolution:

<Accordion title="Column Exclusion">
  Use column exclusion to prevent specific columns from being added to your destination. This is particularly useful for:

  * Excluding sensitive data columns (e.g., PII, passwords)
  * Preventing unnecessary columns from being synced
  * Managing storage costs by excluding large or unused columns

  To use this feature, simply specify the columns you want to exclude in your table's advanced settings.
</Accordion>

<Accordion title="Column Inclusion">
  Column inclusion allows you to explicitly define which columns should be synced to your destination. This feature:

  * Gives you complete control over your destination schema
  * Helps maintain a clean, minimal dataset
  * Requires manual management of new columns

  <Warning>
    Column inclusion is mutually exclusive with column exclusion. When using this feature, you'll need to manually add any new columns to your destination schema as they appear in the source.
  </Warning>
</Accordion>

### Automatic schema changes

A company admin can enable **Automatically apply supported schema changes** from **Settings > Company**. This runs a separate reconciliation job that compares an eligible source table with its existing destination table and applies supported differences.

#### Which sources are supported?

Automatic schema changes apply to sources with authoritative column schemas: **PostgreSQL, CockroachDB, MySQL, PlanetScale, Microsoft SQL Server, Oracle, and Amazon Keyspaces (Cassandra)**. The job also requires a destination that supports column metadata and an existing destination table.

#### When does the job run?

Artie schedules the job daily between **7:00 and 8:00 AM Pacific Time**. It also queues reconciliation after an eligible pipeline redeploys, but not after a pipeline's first deployment. Reconciliation runs asynchronously and does not determine whether a deployment succeeds.

The job only processes eligible running pipelines. It skips locked pipelines and locked data planes, and applies a two-hour cool down per pipeline, so repeated checks or re-deployments do not necessarily start another reconciliation immediately.

#### Does it only add columns?

By default, yes: the job adds source columns that are missing from an existing destination table. It does not rename columns or change the type of an existing destination column.

It can also drop a destination column only when **Enable hard deletes for columns** is enabled in the destination's advanced settings. Column inclusion and exclusion still determine which source columns Artie can add. Dropping columns remains subject to the safety process below.