Selective Backfills by Upstream Source
Table-level backfills in fan-in pipelines can be scoped to selected upstream databases or schemas, with destination merge behavior unchanged.
When multiple databases or schemas fan into a single pipeline, backfills can unintentionally reprocess far more data than intended.
Previously, triggering a backfill at the table level would re-run that table across every upstream database in the pipeline. In environments with 20 or 30 sources, a change intended for one database could execute across all of them.
This update adds selective scoping for table-level backfills in fan-in pipelines.
You can now choose exactly which upstream databases to include in a backfill. For schema-level fan-in, you can scope the backfill to specific schemas. Execution is limited to only the selected sources.
Destination merge behavior remains unchanged. Only the upstream execution scope is constrained.
Why this matters
- Prevents unnecessary reprocessing across all fan-in sources
- Reduces blast radius in multi-tenant pipelines
- Speeds up onboarding of new databases
- Improves operational control during remediation
- Avoids unintended warehouse usage during backfills