MongoDB

Artie syncs data directly from MongoDB change streams, capturing insert, update, and delete operations in real-time.

MongoDB server must be in a replica set. If your pipeline only has a standalone server, you can create a replica set with one member.Need help? Check out this guide.

Using AWS DocumentDB? See our DocumentDB guide instead.

Required settings

Connection string
Service account

Connection string

Go to Atlas UI
Find your pipeline and click “Connect”

We support both MongoDB SRV format or standard connection string.

Retrieve connection string from Atlas

Service account

You can create a service account through the Atlas UI or by running a script.

Option #1 - Atlas UI

Click on Database Access on the left
Click on Add New Database User
Under Database User Privileges, open Built-in Role and Select Only read any database
To limit the user to one database instead, open Specific Privileges, click Add Specific Privilege, and add read on that database and on local. This does not work with one change stream for the whole cluster, which needs readAnyDatabase.

Option #2 - Service account script

/* Read one database. Replace <database> with the database you are replicating. */
const roles = [{ role: "read", db: "<database>" }, { role: "read", db: "local" }];
/* Or read every database. Needed for one change stream for the whole cluster. */
// const roles = ["readAnyDatabase", { role: "read", db: "local" }];

/* If the user does not exist. */
use admin;
db.createUser({ user: "artie", pwd: "<password>", roles });

/* If the user already exists */
db.updateUser("artie", { roles });

Advanced

Enabling full document before change

This feature requires MongoDB 6.0 or later.

If you are replicating a MongoDB collection into a partitioned table downstream, you will want to consider enabling this so that the full document before change is available for deletes.This is because we will need the previous row to grab the partitioned field(s) in order to select the right partition downstream.To enable this, you’ll want to run the following commands:

// Enable preAndPostImage on the replica set
use admin;
db.runCommand({
   setClusterParameter: {
      changeStreamOptions: {
         preAndPostImages: {
            expireAfterSeconds: 100
         }
      }
   }
});

// Enable preAndPostImage on the collection
use databaseName;
db.runCommand({ collMod:"collectionName",changeStreamPreAndPostImages: { enabled: true } });

// See the previous setting
db.adminCommand( { getClusterParameter: "changeStreamOptions" } );

// To disable this behavior, you can set expiredAfterSeconds to off
db.runCommand( {
   setClusterParameter:
      { changeStreamOptions: {
         preAndPostImages: { expireAfterSeconds: 'off' }
      } }
} );

How do high-throughput collections affect other collections?

MongoDB writes every change on a replica set to one oplog. On a sharded cluster, each shard has its own oplog and mongos merges them into one ordered stream. A change stream is a cursor over that oplog.Default: one change stream per pipeline. Artie opens a single database-level change stream for the pipeline. It filters on the server to the collections you selected and the insert, update, replace, and delete operations. Events come back in oplog order across all selected collections.This means a write spike in one selected collection delays the others. If orders takes 10,000 writes per second and users takes one write every few seconds, a users change written after an orders burst is not returned until every orders event ahead of it has been read and processed. Each update also triggers a full document lookup by _id, so a busy collection adds read load as well.Collections you did not select. MongoDB drops their oplog entries on the server, so they are not sent to Artie or looked up. The cursor still has to read past them, so a large write burst anywhere on the replica set adds some latency, but much less than a busy selected collection.Oplog window. The oplog is a capped collection, so heavy writes on any collection shrink how much history it holds. If a pipeline falls further behind than the oplog window, for example during a destination outage or a long pause, its resume token is no longer in the oplog and the affected collections must be backfilled again. Size the oplog for your peak write rate, or set oplogMinRetentionHours on MongoDB 4.4 or later.Isolating a busy collection. You have two options:

Move the busy collection into its own pipeline. It gets its own change stream and its own reader, so spikes there do not delay any other collection.
Enable Use one change stream per collection in the source’s advanced settings. Artie opens a change stream for each selected collection, reads them concurrently, and stores a separate resume token for each. Ordering is preserved within a collection, not across collections. All streams share the same reader. You can only change this setting while the pipeline is in draft or paused. On a pipeline that is already streaming, switching requires backfilling the selected collections, because the existing resume token does not carry over to per-collection streams.

Each change stream reads the oplog separately on the server. Splitting collections stops one from waiting behind another, but it adds read load on the replica set.

How do you handle typing and nested fields?

Data types: Artie determines the data type for each field by looking at the first non-null value and using its BSON type. This type is then mapped to the appropriate destination column type.Mixed-type fields: If the same field contains different BSON types across documents (for example, age: 25 in one document and age: "twenty-five" in another), Artie will land that field as a string in the destination. You can cast it to the desired type downstream.Sparse / dynamic fields: MongoDB collections often have documents with different sets of keys. Artie unions the schema across all observed documents. Fields that are absent from a given document are written as NULL in the destination.Nested objects: Nested documents and arrays are preserved as JSON objects in the destination. Artie does not flatten or unfurl nested structures into separate columns.For example, if the source document contains:

{
  "test_nested_object": {
    "a": {"b": {"c": "hello"}},
    "test_timestamp": {"$timestamp": { "t": 1678929517, "i": 1 }},
    "super_nested": {
      "test_timestamp": {"$timestamp": { "t": 1678929517, "i": 1 }},
      "foo": "bar"
    }
  }
}

Artie will output test_nested_object as:

{
  "a": {
    "b": {
      "c": "hello"
    }
  },
  "super_nested": {
    "foo": "bar",
    "test_timestamp": "2023-03-16T01:18:37Z"
  },
  "test_timestamp": "2023-03-16T01:18:37Z"
}

Note that BSON-specific types (like $timestamp) are converted to standard formats. If the destination does not support a VARIANT or semi-structured type, the nested object will be stored as a JSON string.

Cast all columns to string

MongoDB collections have no enforced schema, so different documents can store the same field with different types. By default, Artie infers the column type from the first non-null value it sees. If a later document stores that field with a different type, Artie lands the column as a string.The Cast all columns to string setting makes this explicit: every column is stored as a string as it is discovered, regardless of the underlying BSON type. Nested objects are stored as JSON strings.This setting applies to new columns as they are discovered. Existing destination columns keep their current types. To cast all columns as strings on a pipeline with an existing destination table, rebuild the table after enabling this setting.You can enable this from your pipeline’s destination advanced settings.

Getting Started

Pipelines

Connectors

Monitoring

Artie Dashboard

Required settings

Connection string

Service account

Advanced

​Required settings

​Connection string

​Service account

​Advanced

Required settings

Connection string

Service account

Advanced