History Table Backfill
New pipelines backfill the full current state of a table into its history table before streaming ongoing changes.
History tables are most useful when they’re complete. Previously, Artie’s history tables captured changes going forward - which worked well for many workflows, but left teams filling the initial gap with custom scripts or heavy batch jobs when they needed a full baseline. That added setup complexity and made it harder to rely on history tables for downstream incremental processing from day one.
Artie can now generate a full historical snapshot for history tables before streaming incremental changes.
When you set up a new pipeline, Artie backfills the entire current state of a table into the history table, then seamlessly transitions to streaming ongoing changes. This ensures the history table starts complete, consistent, and ready for production use.
Why this matters:
- Start history tables with a complete and accurate baseline
- Eliminate custom backfill scripts and batch jobs
- Enable incremental models and continuous processing from day one
- Improve consistency for audits, time-travel queries, and analytics