Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Using Artie

Column inclusion and exclusion

Learn how column inclusion and exclusion works in Artie.

Overview

FeatureDescription
Column InclusionReplicate only the columns you specify
Column ExclusionReplicate all columns except the ones you exclude
Should I use column inclusion or exclusion?

Column Inclusion and Exclusion

One source, several destinations

A shared source reader can feed several pipelines, each with its own column settings. It’s set up in the Terraform provider; see Write to multiple destinations.

Architecture

Source ReaderPipeline Table 1Pipeline Table 2

Each pipeline table will have its own Kafka consumer group and we can specify additional column inclusion and exclusion for each pipeline table.

Where should I specify the column inclusion and exclusion?

You can specify column inclusion and exclusion at the source reader and pipeline table level. The diagram below shows where to set each one.