Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Destinations

Redis destination connector: setup and configuration

Configure Redis as a destination in Artie. Artie writes each table's changes to a Redis Stream as JSON entries, with support for TLS, ACL users, SSH tunnels, Redis Cluster, and an optional cap on stream length.

What you’ll need

  • Redis 5.0 and later, which added Redis Streams. To sign in with a username, Redis 6.0 and later, which added ACLs.
  • Network access from Artie to your Redis host and port: a direct connection that allows Artie’s IP addresses, or an SSH tunnel.
  • If your Redis uses ACLs, a user that can run XADD and DEL on Artie’s stream keys. You create it in the first step.

Setup

  1. Create a Redis user for Artie

    Artie writes to keys that start with redis:<namespace>:. The namespace is public unless you set one on the pipeline. Replace public in the commands below if you use a different namespace. Artie needs DEL to drop a table’s stream, for example before a backfill set to Drop and recreate.

    Bash
    redis-cli ACL SETUSER artie on '>s3cure-Passw0rd' '~redis:public:*' +@connection +xadd +del

    If the user is missing a permission, Redis rejects the command and the connection test or the write fails with a NOPERM error, for example NOPERM User artie has no permissions to run the 'cluster|slots' command or NOPERM No permissions to access a key. Add the missing command or key pattern to the user.

    If your Redis only uses a password (requirepass), skip this step and leave Username (optional) blank in the next step.

  2. Add the Redis connector in Artie

    Create a destination connector, choose Redis, and fill in:

    SettingWhat to enter
    HostThe Redis hostname or IP address. For Redis Cluster, use the configuration endpoint.
    PortThe Redis port. Defaults to 6379.
    Username (optional)The ACL user from the previous step. Leave it blank for password-only Redis.
    Password (optional)The user’s password, or the requirepass password.
    Cluster modeTurn this on for Redis Cluster, such as ElastiCache with cluster mode enabled.
    Database numberThe logical database to write to. Defaults to 0. Hidden when Cluster mode is on.
    Max stream lengthRoughly how many entries each table’s stream keeps. Defaults to 0, which keeps every entry. See Stream length and trimming.
    Enable TLS/SSLTurn this on if your Redis requires TLS. Artie uses TLS 1.2 or later and verifies the server certificate against Host.

    Redis Cluster only has database 0. If Cluster mode is on and Database number isn’t 0, saving fails with Redis Cluster only supports database 0.

    For the connection method, choose a direct connection or an SSH tunnel:

    • SSH tunnel works for standalone Redis. With TLS on, Artie still verifies the certificate against Host, not the tunnel. Redis Cluster can’t use a tunnel, because the cluster tells clients to connect to each node’s own address, which the tunnel doesn’t forward. Saving a cluster connector with a tunnel fails with SSH tunnels aren't supported for Redis Cluster.
    • PrivateLink isn’t supported for Redis. Saving fails with PrivateLink isn't supported for Redis.
  3. Set the namespace on your pipeline

    In the pipeline’s destination settings, set Namespace (optional) to group this pipeline’s streams under one prefix. It defaults to public.

    You can’t change the namespace while the pipeline is running. Pause the pipeline first, then rename the existing stream keys to the new prefix, or Artie starts writing to new streams and your consumers miss the old ones.

Advanced

Stream keys and entry format

Artie writes each table to its own Redis Stream with the key redis:<namespace>:<table>, for example redis:public:orders.

Each stream entry has one field, data, holding the row as a JSON object keyed by column name:

Bash
redis-cli XRANGE redis:public:orders - + COUNT 1
Text
1) 1) "1791941136611-0"
   2) 1) "data"
      2) "{\"__artie_delete\":false,\"__artie_operation\":\"c\",\"id\":1042,\"price\":\"19.99\",\"status\":\"pending\"}"
  • Every entry includes __artie_operation: c for create, u for update, d for delete, and r for rows from a backfill. Artie always adds this column for Redis, so the Add ‘Artie operation’ column setting doesn’t apply.
  • __artie_delete is true when the row was deleted in the source.
  • Decimal values are JSON strings, such as "19.99", so no precision is lost.
  • Other system columns appear when you turn them on in the pipeline’s advanced settings.

To read the stream with a consumer group:

Bash
redis-cli XGROUP CREATE redis:public:orders analytics 0 MKSTREAM
redis-cli XREADGROUP GROUP analytics worker-1 COUNT 100 STREAMS redis:public:orders '>'
Changes to the same row within a flush

Artie buffers changes and writes them when a flush rule triggers. Within one flush, it keeps only the latest change for each primary key. If a row is updated three times between flushes, the stream gets one entry with the final values. If a row is created and then deleted between flushes, the stream gets only the delete.

Shorter flush rules write intermediate changes more often, but don’t guarantee one entry per source change.

Delivery guarantees and retries

Delivery is at least once. Make your consumers idempotent, for example by upserting on the primary key.

  • If a write to Redis fails, for any reason, Artie retries it up to 15 times, waiting between 1 and 30 seconds between attempts. If every attempt fails, Artie reports a replication error, restarts, and tries again from its last committed position.
  • Artie only commits its position after a write succeeds, so a failed write doesn’t lose changes. They’re written once Redis accepts writes again.
  • Artie sends each flush’s entries to Redis in one pipeline, which isn’t a transaction. If a write fails partway through, the retry writes the entries that already landed again, and consumers see them twice.
Stream length and trimming

Max stream length caps each table’s stream. Artie passes it to XADD as MAXLEN ~, so Redis trims old entries as Artie adds new ones. Trimming is approximate: Redis only removes whole internal nodes, so a stream can run somewhat over the cap. The cap applies to every stream the connector writes.

With the default of 0, Artie never trims, and each stream grows until you trim it with XTRIM. If Redis reaches maxmemory and can’t evict, it rejects writes with an OOM error. Artie keeps retrying as described in Delivery guarantees and retries, and the pipeline makes no progress until you free memory or raise maxmemory.

Redis Cluster

With Cluster mode on, Artie connects to the configuration endpoint, reads the cluster’s slot map, and writes each stream to the node that owns its key. Each flush for a table writes to a single key, so all of its entries go to one node.

If Cluster mode is off and Host points at a cluster, Redis answers writes to keys on other nodes with MOVED <slot> <host>:<port>. Retrying doesn’t help, so the pipeline reports a replication error that contains MOVED. Turn on Cluster mode to fix it.

Limitations

  • PrivateLink isn’t supported. Saving the connector with a PrivateLink connection fails. Connect directly or through an SSH tunnel.
  • Redis Cluster can’t use an SSH tunnel. Saving fails. Connect to the cluster directly and allow Artie’s IP addresses.
  • Redis Cluster writes to database 0 only. This is a Redis Cluster restriction. Use the namespace to separate pipelines.
  • One stream length cap per connector. Max stream length applies to every table. To use different caps, create a separate Redis connector for those tables.
  • Not every source change becomes an entry. Changes to the same row within one flush collapse into the latest. See Changes to the same row within a flush.

Frequently asked questions

Why are entries missing from my stream?

Redis trimmed them, or Artie collapsed them within a flush. Check Max stream length against how far your consumers fall behind, and see Changes to the same row within a flush.

Why do my consumers see the same entry twice?

A write failed partway and Artie retried it. Delivery is at least once, so make consumers idempotent. See Delivery guarantees and retries.

How do I tell a delete from an update?

Read __artie_operation. It’s d for deletes and u for updates, and every entry has it.