Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Destinations

Google Cloud Storage

How to configure Google Cloud Storage (GCS) as an append-only Parquet destination in Artie.

Required settings

  • Project ID
  • Service account
  • Bucket name

Optional settings

  • Optional prefix
  • Table name separator (default: “/”) Artie will write the fully qualified table name into the bucket (e.g: db/schema/table), you can control the separator between the parts

Bucket structure

Artie writes append-only Parquet files to your GCS bucket. The files use GZIP compression. Artie does not create a Delta transaction log.

Text
-- tableName is db, schema and table joined with the table name separator (default: db/schema/tableName)
/{{bucketName}}/[{{prefix}}/]{{tableName}}/date={{YYYY-MM-DD}}/{{unixTimestampMs}}_{{randomString(4)}}.parquet

-- example
/artie/foo/db/schema/tableName/date=2026-08-21/1654320000000_abc1.parquet
Service account script
HCL
provider "google" {
  project = "your-gcp-project-id"
  region  = "us-central1"
}

resource "google_service_account" "artie_transfer" {
  account_id   = "artie-transfer"
  display_name = "Artie Transfer Service Account"
  description  = "Service account for Artie to write to GCS"
}

resource "google_storage_bucket_iam_member" "bucket_object_admin" {
  bucket = "your-bucket-name"
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.artie_transfer.email}"
}

resource "google_storage_bucket_iam_member" "bucket_viewer" {
  bucket = "your-bucket-name"
  role   = "roles/storage.bucketViewer"
  member = "serviceAccount:${google_service_account.artie_transfer.email}"
}

resource "google_service_account_key" "artie_transfer_key" {
  service_account_id = google_service_account.artie_transfer.name
}

output "service_account_email" {
  value = google_service_account.artie_transfer.email
}

output "service_account_key" {
  value     = base64decode(google_service_account_key.artie_transfer_key.private_key)
  sensitive = true
}