Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Sources

DocumentDB

Learn how to use Artie to replicate data from DocumentDB via change streams.

Required settings

  • Cluster endpoint
  • Port (default is 27017)
  • Service account
  • Database name (ensure the database has change stream enabled)
Need help finding the cluster endpoint?

You can find the cluster endpoint by going to the AWS console, selecting the DocumentDB cluster and go to the “Configuration” tab.

DocumentDB cluster endpoint

Setting up change streams and a service account

DocumentDB keeps change stream events for 3 hours by default. If a pipeline is paused or behind for longer, its resume token expires and the affected collections must be backfilled again. Raise change_stream_log_retention_duration in the cluster parameter group (up to 7 days) to cover how long you might pause the pipeline. See Using change streams with Amazon DocumentDB.

setup.js
// Creating a service account for Artie to subscribe to change stream
db.createUser({user: "artie", pwd: "changeme", roles: ["read"]});
// Grants access to DocumentDB change streams
use admin;
db.grantRolesToUser("artie", [
  { role: "readAnyDatabase", db: "admin" }
]);

// Depending on the permission granularity you want, 
// you can use the following commands to grant permissions to the service account.
// Enable change streams for all collections in database "changeme"
db.adminCommand({
    modifyChangeStreams: 1,
    database: "changeme",
    collection: "",
    enable: true
});

// Enable change streams for all collections in all databases
db.adminCommand({
    modifyChangeStreams: 1,
    database: "",
    collection: "",
    enable: true
});

// Optional - Enable change streams for the collection "foo" in database "changeme"
db.adminCommand({
    modifyChangeStreams: 1,
    database: "changeme",
    collection: "foo",
    enable: true
});

Advanced

Cast all columns to string

DocumentDB collections have no enforced schema, so different documents can store the same field with different types. By default, Artie infers the column type from the first non-null value it sees. If a later document stores that field with a different type, Artie lands the column as a string.

The Cast all columns to string setting makes this explicit: every column is stored as a string as it is discovered, regardless of the underlying BSON type. Nested objects are stored as JSON strings.

This setting applies to new columns as they are discovered. Existing destination columns keep their current types. To cast all columns as strings on a pipeline with an existing destination table, rebuild the table after enabling this setting.

You can enable this from your pipeline’s destination advanced settings.