Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Tables

System columns

Artie adds system columns to your destination tables for change tracking, soft deletes, and source metadata.

Artie can add columns prefixed with __artie_ to your destination tables. These columns track how and when each row was changed. You can configure them in your pipeline’s destination Advanced settings or under table settings.

Artie operation column

In normal mode, enable Add ‘Artie operation’ column in your pipeline’s destination Advanced settings to add __artie_operation to the destination table. The column identifies the operation that most recently produced the row.

In history mode, __artie_operation is included in each history record and identifies the operation that produced that record.

ValueMeaning
rBackfill
cCreate
uUpdate
dDelete

History mode columns

Artie adds these columns to history mode tables automatically. To add them to normal tables, turn on Add ‘Artie updated at’ column or Add ‘database updated at’ column in your destination’s Advanced settings, under Additional data. Both are timezone-aware timestamps, for example TIMESTAMP_TZ in Snowflake.

ColumnData typeDescription
__artie_updated_atTIMESTAMP_TZThe timestamp of when Artie processed the change.
__artie_db_updated_atTIMESTAMP_TZThe timestamp of when the change occurred in your source database.

Soft delete columns

These columns are added when you turn on Soft-delete rows in your destination’s Advanced settings. Instead of removing deleted rows, Artie flags them so you can retain the data.

ColumnData typeDescription
__artie_deleteBOOLEANtrue if the row was deleted in the source database.
__artie_only_set_deleteBOOLEANS3 and GCS destinations only. true if the row was deleted and the event contains no other column values to update, only the delete flag itself.
What is the difference between __artie_delete and __artie_only_set_delete?

Both columns indicate a deletion, but they serve different purposes:

  • __artie_delete is set to true whenever a row is deleted.
  • __artie_only_set_delete is set to true when the delete event carries no additional column values, meaning there is nothing to update in the destination besides the delete flag.

For example, if a row is updated and then deleted within the same flush cycle, the event contains both the updated values and the deletion. In that case, __artie_delete = true and __artie_only_set_delete = false.

Static columns

Static columns let you add custom key-value pairs to every row written to your destination table. Each static column has a fixed name and value that you define. The value does not change per row.

This is useful when you need to tag rows with metadata that isn’t present in the source, such as an environment label, a data source identifier, or a tenant ID.

Add static columns in your destination’s Advanced settings, under Additional data > Static columns.

Source metadata columns

These columns provide context about where each row originated and the underlying database event that produced it. Turn them on in your destination’s Advanced settings, under Additional data: Add ‘full source table name’ column and Add ‘source metadata’ column.

ColumnData typeDescription
__artie_full_source_table_nameSTRINGThe fully qualified source table name in the format dbName.schemaName.tableName.
__artie_source_metadataJSONSource metadata from the change event, such as the log sequence number (LSN), transaction ID, and other database-specific fields.