Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Destinations

S3

How to configure S3 as an append-only Parquet destination in Artie.

Required settings

  • Bucket name
  • AWS region
  • Credentials: static access keys (Static credentials) or an IAM role Artie assumes (Role assumption)

Optional settings

  • Folder name
  • Table name separator (default: /)

Bucket structure

Artie writes append-only Parquet files to your S3 bucket. It does not create a Delta transaction log.

Text
-- tableName is db, schema and table joined with the table name separator (default: db/schema/tableName)
/{{bucketName}}/[{{folderName}}/]{{tableName}}/date={{YYYY-MM-DD}}/{{unixTimestampMs}}_{{randomString(4)}}.parquet

-- example
/artie/foo/db/schema/tableName/date=2026-08-21/1654320000000_abc1.parquet
Service account script
HCL
provider "aws" {
  region = "us-east-1" # Your AWS Region
}

resource "aws_iam_user" "artie_transfer" {
  name = "artie-transfer" # Your service account name
}

resource "aws_iam_policy" "s3_read_write_policy" {
  name        = "s3-read-write-policy"
  description = "Ability to list, read and write to the bucket"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket",
          "s3:GetEncryptionConfiguration"
        ]
        Resource = "arn:aws:s3:::your-bucket-name"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject"
        ]
        Resource = "arn:aws:s3:::your-bucket-name/*"
      },
      // If the bucket is encrypted with KMS:
      {
        Effect = "Allow",
        Action = [
          "kms:GenerateDataKey",
          "kms:Decrypt",
        ],
        Resource = [
          "arn:aws:kms:your-region:your-account-id:key/your-kms-key-id",
        ]
      }
    ]
  })
}

resource "aws_iam_user_policy_attachment" "user_policy_attachment" {
  user       = aws_iam_user.artie_transfer.name
  policy_arn = aws_iam_policy.s3_read_write_policy.arn
}

resource "aws_iam_access_key" "artie_transfer_key" {
  user    = aws_iam_user.artie_transfer.name
}

output "access_key_id" {
  value     = aws_iam_access_key.artie_transfer_key.id
  sensitive = false
}

output "secret_access_key" {
  value     = aws_iam_access_key.artie_transfer_key.secret
  sensitive = true
}
Role assumption trust policy

To use Role assumption, create an IAM role, attach the s3-read-write-policy from the service account script to it, and enter its AWS role ARN in Artie. Add this trust policy to the role so Artie can assume it. On a data plane you run yourself, trust the IAM identity your data plane runs as instead of Artie’s principal.

The external-ID condition is optional. Include it only when you enter an AWS external ID in Artie, and replace my-external-id with that exact value.

JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::788826912768:user/ci"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "my-external-id"
        }
      }
    }
  ]
}