Getting started
Architecture
Learn about Artie's split plane architecture that separates control and data planes for enhanced security and flexibility.
Artie follows a split plane architecture where the control plane and data plane are separated:
- Control plane: Dashboard, pipeline configuration, monitoring, and metrics.
- Data plane: All data processing (reading from your source, buffering changes, and writing to your destination). No customer data leaves the data plane.
Only metadata (table names, column schemas, pipeline status) is sent from the data plane to the control plane.
Where your data flows
Pick a deployment, then select any part of the pipeline.
Cloud
Artie runs both planes, with the data plane in the same region as your databases to keep egress and latency low. Data flows from your source, through Artie's data plane, to your destination.
How data flows
Artie replicates database changes in near real-time using a reader-buffer-writer pattern:
- Change capture (reader) - Artie connects to your source database and subscribes to its transaction log to capture changes. During backfills, a separate process scans full tables and writes directly to the destination while CDC continues in parallel.
- Buffering (Kafka) - Each change event is published to an internal Kafka queue. This buffer enables parallelism and absorbs throughput spikes when downstream systems are slow.
- Flush to destination (writer) - When a flush condition is met, Artie writes the buffered data to the destination using idempotent upserts or merges, commits the Kafka offset, and resumes reading.
- Metadata to control plane - Only pipeline configuration, status, and metrics are sent to the control plane. No row-level data is transmitted.
Deployment options
Cloud
Both the control plane and data plane run in Artie’s cloud infrastructure. The data plane is co-located in the same region as your databases to minimize egress costs and latency.
- Fully managed with minimal operational overhead
- Data flows from your source, through Artie’s data plane, to your destination
BYOC
The data plane runs in your cloud account or private network. Artie hosts the control plane and manages upgrades and scaling.
You host the data plane infrastructure (Kubernetes, Kafka, persistent storage), and Artie runs operations remotely.
- All database connections originate inside your environment
- Production data never leaves your VPC
- Control plane communicates with the data plane over HTTPS via authenticated endpoints
Security
Networking
Artie supports multiple methods for connecting to your sources and destinations:
- PrivateLink - Traffic stays on the cloud provider’s private network with no public internet exposure. Supported on AWS (PrivateLink) and Azure (Private Link).
- IP allowlisting - Artie provides fixed IP ranges that you allowlist on your database firewall.
- SSH tunnels - Artie connects to a bastion host you run, which forwards traffic to your database. Allow inbound SSH from Artie’s IPs on the bastion.
For setup steps for each method, see Connection options.
Encryption
- In transit: Traffic between planes is encrypted with TLS. Connections to your databases use TLS when it’s enabled in the connection’s settings.
- At rest: Data at rest is encrypted with AES-256.