Backfill Tuning: Picking the Right Batch Size
Backfill batch size is now configurable, and the default rises from 5,000 to 25,000 rows per chunk.
You can now control how many rows Artie processes at a time during backfills. The default is now 25,000 rows per chunk (up from 5,000), but you can tune this based on performance vs. load tradeoffs.
Why backfill batch size matters:
Backfills aren’t one-size-fits-all. Some teams want speed. Others are sensitive to database load and tiptoeing around a production DB at 2am. Until now, everyone got the same batch size of 5,000 rows per chunk. Now you can tune backfills to match your style:
- The default is 25,000 rows - we benchmarked a bunch of sizes. 25,000 rows won out. So that’s our new default
- You have control - adjust the batch size to fit your environment
How to tune batch size based on your workload:
- Speed up backfills: Larger chunks = fewer queries, can improve throughput, but overly large chunks can backfire. It’s about finding balance.
- Reduce DB load: Smaller chunks = faster queries, lower impact on source
Need help tuning batch size?
If you're unsure what batch size is right for your workload, reach out - we'll help you tune it.