Advanced Settings
To find table settings, click on your pipeline > Edit > Tables > Table settings
BigQuery partitioning
BigQuery partitioning
If your BigQuery table is partitioned, you can specify the partitioned columns here. Click here for instructions on how to partition your table in BigQuery.
Snowflake merge predicate
Snowflake merge predicate
Artie merges data into your Snowflake based on your source table’s primary key(s). If you have an additional cluster key defined, you can specify that here.
For example, if the table was clustered by created_at, you can specify that column here.
Use different name in destination
Use different name in destination
This setting allows you to define a new table alias. If your source table is named foo, you can name it bar.
Skip deletes
Skip deletes
When enabled, Artie will skip processing deletes. This setting is useful if you want to keep a smaller dataset in your source and have a complete archive in your destination.
History Tables
History tables provide a complete audit trail of all changes to your data. When enabled, Artie creates a separate table named {TABLE}__HISTORY that records every change made to the original table.
The history table includes these additional columns:
__artie_operation- The type of change (CREATE, UPDATE, or DELETE)__artie_updated_at- When Artie processed the change__artie_db_updated_at- When the change occurred in your source database
What do people use history tables for?
What do people use history tables for?
Here are some examples of what you can do with history tables:
- Tracking data changes over time
- Auditing and compliance.
- Data recovery and point-in-time analysis.
How do I enable history tables?
How do I enable history tables?
- Edit your pipeline
- Go into the
Tablestab - Find the table(s) you want to enable history mode