Overview

How it works

Flushing Flushing is the process of loading buffered data from memory into your destination database. is a critical part of Artie’s data pipeline that determines when and how data gets written to your destination.

1. Data buffering

   • Artie’s reading process will read changes from your source database and publish them to Kafka
   • Artie’s writing process will read messages from Kafka and write them to your destination
   • Messages are temporarily stored in memory and deduplicated based on primary key(s) or unique index
   • Multiple changes to the same record are merged to reduce write volume

2. Flush trigger evaluation

   • Artie continuously checks three conditions that you control
   • It also flushes early if it’s running low on memory, which shows as memory in the Flush count chart
   • When any condition is met, a flush is triggered
   • Reading from Kafka pauses during the flush operation

3. Data loading

   • Buffered data is written to your destination in an optimized batch
   • After completion, Artie will commit the offset and resume reading from Kafka
   • The cycle repeats for continuous data flow

Set flush rules

Open your pipeline, click Edit pipeline, and go to the Destination tab. Under Flush rules, pick a preset or choose Custom:

With Custom, time elapsed can be 5 seconds to 6 hours, changed records must be at least 5, and batch size can be up to 1.5 GB.

Conditions

Artie evaluates three conditions to determine when to flush data. Any one of these conditions will trigger a flush: