Write to multiple destinations

Overview

Use a shared source reader when you want one source database to feed multiple destinations without creating a second CDC reader. Artie captures the source change stream once and each pipeline consumes it independently.

This guide starts with an existing PostgreSQL to Snowflake pipeline and adds BigQuery as a second destination. The same pattern works for other supported source and destination combinations.

What you’ll need

• An existing pipeline and source reader managed in Terraform, or imported into Terraform state.
• A configured destination connector for each destination. This guide uses snowflake_analytics and bigquery_lake.
• The full list of source tables required by either destination.
• Permission to run terraform plan and terraform apply.

How it works

Before the change, the source reader is dedicated to one pipeline:

After the change, the same reader publishes one change stream and both pipelines consume it independently:

Each pipeline owns its destination configuration, table settings, backfill, and stream position. Changing a pipeline’s destination settings does not change the other pipeline. The shared source reader owns which source tables Artie captures.

Steps

1. Make the existing source reader shared

   Add is_shared = true to the existing source reader and declare the source tables it must capture. The reader’s tables map must be a superset of the tables in every pipeline that references it.

   terraformCopyCopied

   resource "artie_source_reader" "postgres_production" {
     name                               = "Postgres Production Reader"
     connector_uuid                     = artie_connector.postgres_production.uuid
     database_name                      = "app"
     postgres_replication_slot_override = "artie_reader"
   
     is_shared = true
   
     # Capture every table needed by either pipeline.
     tables = {
       "public.customers" = {
         name   = "customers"
         schema = "public"
       }
       "public.orders" = {
         name   = "orders"
         schema = "public"
       }
     }
   }

   Preserve the source reader’s existing connection and source-specific settings. Run terraform plan and confirm this step changes only the source reader, then apply it.

2. Verify the existing pipeline

   The existing pipeline already references this source reader, so promoting the reader to shared does not require a pipeline change. Keep the pipeline’s current destination configuration and table selection.

   Confirm that its source_reader_uuid still references artie_source_reader.postgres_production.uuid. Do not copy or reapply the pipeline resource solely to make the reader shared, especially when the pipeline was imported into Terraform.

3. Add a second pipeline for the new destination

   Create another artie_pipeline that references the same source_reader_uuid and a different destination connector. A pipeline may replicate only a subset of the reader’s captured tables.

   terraformCopyCopied

   resource "artie_pipeline" "postgres_to_bigquery" {
     name               = "PostgreSQL to BigQuery"
     source_reader_uuid = artie_source_reader.postgres_production.uuid
   
     destination_connector_uuid = artie_connector.bigquery_lake.uuid
     destination_config = {
       dataset = "product_analytics"
     }
   
     # BigQuery receives orders only. The shared reader still captures
     # customers because the Snowflake pipeline needs it.
     tables = {
       "public.orders" = {
         name   = "orders"
         schema = "public"
       }
     }
   }

   Run terraform plan and verify that Artie will create only the new pipeline. Then run terraform apply.

   Artie backfills the new pipeline’s tables into BigQuery and then consumes live changes from the shared reader. The existing Snowflake pipeline continues from its own position throughout the backfill.

4. Validate both destinations

   Wait for the new pipeline’s backfill to complete and its replication lag to reach your normal operating range in the analytics portal.

   Validate the new destination before relying on it:

   • Compare row counts for each newly replicated table.
   • Spot-check recent inserts, updates, and deletes.
   • Run representative downstream queries against both destinations.
   • Confirm that the existing pipeline remains healthy while the new pipeline catches up.

Add another table later

When adding a table to one of the pipelines, add it to the shared reader’s tables map first if it is not already present. The reader must capture a table before any attached pipeline can consume it.

For example, to send public.refunds only to BigQuery:

1. Add public.refunds to artie_source_reader.postgres_production.tables.
2. Apply the source reader change.
3. Add public.refunds to artie_pipeline.postgres_to_bigquery.tables.
4. Apply the pipeline change and monitor its backfill.

Frequently asked questions

Does this create another replication slot or source reader?

No. Both pipelines reference the same artie_source_reader, so Artie captures the source change stream once and fans it out to the attached pipelines.

Can the two pipelines replicate different tables?

Yes. Each pipeline has its own tables map. The shared source reader’s map must include the union of all tables used by every attached pipeline.

Can the pipelines use different table settings?

Yes. Destination-specific settings, such as an alias, history mode, column inclusion, and destination configuration, belong to each pipeline. Source capture settings belong to the shared reader and apply to every attached pipeline.

Can I add more than two destinations?

Yes. Add another artie_pipeline that references the same shared reader. Keep the reader’s tables map aligned with the union of all attached pipelines.

Can I switch the reader back to a dedicated reader later?

Yes, but only after at most one pipeline remains attached. Set is_shared = false, remove the shared reader’s tables map, and apply. See online migration for the full conversion steps.