Skip to main content

​
Introduction

Below, you can see sample configuration files to describe different workloads. To see all the available settings, please see the options page.

​
Postgres options

​
Postgres -> Snowflake

outputSource: snowflake
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.postgres.wal2json
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
snowflake:
  account: ACCOUNT_ID
  username: USER
  password: PASSWORD
  warehouse: DWH_NAME
  region: us-east-2.aws

​
Postgres -> BigQuery

outputSource: bigquery
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.postgres.wal2json
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
bigquery:
  pathToCredentials: PATH_TO_CREDENTIALS
  projectID: PROJECT_ID
  defaultDataset: DEFAULT_DATASET

​
MongoDB options

​
MongoDB -> Snowflake

outputSource: snowflake
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.mongodb
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
snowflake:
  account: ACCOUNT_ID
  username: USER
  password: PASSWORD
  warehouse: DWH_NAME
  region: us-east-2.aws

​
MongoDB -> BigQuery

outputSource: bigquery
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.mongodb
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
bigquery:
  pathToCredentials: PATH_TO_CREDENTIALS
  projectID: PROJECT_ID
  defaultDataset: DEFAULT_DATASET

​
Optional Blocks

​
Error Reporting to Sentry

If you provide your Sentry DSN; Artie Transfer will automatically report any errors during processing into your Sentry project. Visit this link to see how you can find your Sentry DSN. 
reporting:
  sentry:
    dsn: https://docs.sentry.io/product/sentry-basics/dsn-explainer/
Transfer is using a vendor neutral logging library, so file a feature request if you use Rollbar, or another vendor!

​
Telemetry

Click here to see all the metrics that Transfer emits. 
telemetry:
  metrics:
    provider: datadog
    settings:
      tags:
       - env:production
       - customer:artie.so
      namespace: "transfer."
      addr: "127.0.0.1:8125"
⌘I