Destinations
S3
How to configure S3 as an append-only Parquet destination in Artie.
Required settings
- Bucket name
- AWS region
- Credentials: static access keys (Static credentials) or an IAM role Artie assumes (Role assumption)
Optional settings
- Folder name
- Table name separator (default:
/)
Bucket structure
Artie writes append-only Parquet files to your S3 bucket. It does not create a Delta transaction log.
Text
-- tableName is db, schema and table joined with the table name separator (default: db/schema/tableName)
/{{bucketName}}/[{{folderName}}/]{{tableName}}/date={{YYYY-MM-DD}}/{{unixTimestampMs}}_{{randomString(4)}}.parquet
-- example
/artie/foo/db/schema/tableName/date=2026-08-21/1654320000000_abc1.parquetService account script
HCL
provider "aws" {
region = "us-east-1" # Your AWS Region
}
resource "aws_iam_user" "artie_transfer" {
name = "artie-transfer" # Your service account name
}
resource "aws_iam_policy" "s3_read_write_policy" {
name = "s3-read-write-policy"
description = "Ability to list, read and write to the bucket"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"s3:ListBucket",
"s3:GetEncryptionConfiguration"
]
Resource = "arn:aws:s3:::your-bucket-name"
},
{
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:PutObject"
]
Resource = "arn:aws:s3:::your-bucket-name/*"
},
// If the bucket is encrypted with KMS:
{
Effect = "Allow",
Action = [
"kms:GenerateDataKey",
"kms:Decrypt",
],
Resource = [
"arn:aws:kms:your-region:your-account-id:key/your-kms-key-id",
]
}
]
})
}
resource "aws_iam_user_policy_attachment" "user_policy_attachment" {
user = aws_iam_user.artie_transfer.name
policy_arn = aws_iam_policy.s3_read_write_policy.arn
}
resource "aws_iam_access_key" "artie_transfer_key" {
user = aws_iam_user.artie_transfer.name
}
output "access_key_id" {
value = aws_iam_access_key.artie_transfer_key.id
sensitive = false
}
output "secret_access_key" {
value = aws_iam_access_key.artie_transfer_key.secret
sensitive = true
}Role assumption trust policy
To use Role assumption, create an IAM role, attach the s3-read-write-policy from the service account script to it, and enter its AWS role ARN in Artie. Add this trust policy to the role so Artie can assume it. On a data plane you run yourself, trust the IAM identity your data plane runs as instead of Artie’s principal.
The external-ID condition is optional. Include it only when you enter an AWS external ID in Artie, and replace my-external-id with that exact value.
JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::788826912768:user/ci"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "my-external-id"
}
}
}
]
}