Skip to main content
Use Amazon S3 as a source when an upstream system lands files in S3. This guide walks you through configuring the source in the Artie Dashboard.
Amazon S3 sources are currently controlled by a feature flag. Ask your Artie contact to enable S3 sources for your company before creating an S3 source connector.

​
What you’ll need

  • An S3 bucket in the AWS Region that contains the files to ingest
  • Permission to create an IAM user or IAM role in the AWS account that owns the bucket
  • A destination configured in the Dashboard
  • A decision about whether Artie should process files already in the bucket when the source first starts

​
Configure an S3 source in the Dashboard

1

Create the source

In the Dashboard, create a pipeline, choose Amazon S3 as the source, and configure the destination before adding path mappings.
2

Choose how Artie authenticates to AWS

Under the Source tab, choose one of these authentication methods:
Create an IAM user and access key, then enter the AWS access key ID and AWS secret access key in the Dashboard.
3

Grant bucket permissions

Attach a policy to the IAM user or role that permits Artie to list buckets, list the selected bucket, and read its objects. Replace acme-ingestion with your bucket name.
For role assumption, also add this trust policy to the role. The external-ID condition is optional: include it only when you entered an AWS external ID in the Dashboard, and replace my-external-id with that exact value.
4

Select the bucket and Region

In the Source tab, select the AWS region and then the bucket. The bucket selector lists buckets available to the configured credentials.
5

Add path mappings

Open Path mappings and add a mapping for each group of files that should write to a destination table. For each mapping, choose the S3 prefix, optional filename match, destination table, parser, and primary key.Path mappings are the source of truth for destination tables. Nothing is replicated until at least one mapping exists.
6

Save, deploy, and verify the first file

Save and deploy the pipeline. Upload a file that matches one mapping, then confirm that its rows arrive in the selected destination table before relying on the source for production data.

​
Configure path mappings

A path mapping routes matching S3 objects to one destination table. You can add several mappings, including several that write to the same table.
  • Prefix: Limits the mapping to a portion of the selected bucket.
  • Filename match: Optionally filters by a regular expression applied to the base filename, after a compression suffix is removed. The prefix is not part of this match. For example, use ^orders-[0-9]+\.csv$ to match incoming/orders/orders-42.csv; do not include incoming/orders/ in the expression.
  • Destination table: Selects where the mapping’s rows land.
  • Parser: Choose csv, json, parquet, or ach. CSV mappings can configure the delimiter, comment prefix, headers, whitespace trimming, and values that represent NULL. Gzip-compressed objects are detected and decompressed automatically.
  • Primary key: For recurring snapshot files, specify stable key columns such as id. Artie uses them for destination merge semantics. Without a primary key, Artie uses the source path and record number, which is appropriate for append-style files but not snapshots that must update existing rows.
Keep each mapping after it begins processing. Artie uses a mapping’s internal identity to retain its progress; deleting and recreating a mapping causes it to start with a new offset.

​
File requirements

  • CSV: Files need a header row unless you provide the header names in the mapping. Every data row must have the same number of fields as the header.
  • JSON: Use either one top-level array of objects or newline-delimited JSON (NDJSON), with one object per line.
  • Parquet: Use a flat schema. Nested structs, maps, and lists of structs are not supported.
  • ACH: Artie writes one row for each ACH entry-detail record. A structurally invalid file is rejected as a whole.

​
What happens after deployment

Artie polls the configured path-mapping prefixes on the configured interval. It reads an object only after the settling delay, so a newly uploaded object has time to finish before processing begins. When the source first starts, Artie begins from new objects by default. Enable the source’s existing-object option when you want it to ingest objects that were already present under the configured prefixes. Each mapping retains its own checkpoint, so one mapping can progress independently of the others. Artie advances a checkpoint only after it has processed the object’s rows and flushed that mapping successfully. If processing fails, the checkpoint does not advance for that mapping. S3 ingestion is at least once. Artie includes objects at a checkpoint boundary again to avoid missing files that share the same S3 modification timestamp. Configure a stable primary key when duplicate processing must update an existing destination row rather than create another row.

​
Snapshots and deletes

Artie ingests each file as rows; it does not compare one snapshot file with the previous one to infer deletes. A stable primary key lets subsequent snapshots update matching destination rows, but rows that disappear from a later file are not removed automatically. Handle deletions separately when the destination must mirror those removals.

​
Troubleshooting the first file

If the test file does not arrive in the destination table, confirm that it is in the selected bucket and Region, its key matches the mapping prefix and optional filename pattern, and it has passed the settling delay. Confirm the IAM identity has the listed S3 permissions. For objects encrypted with a customer-managed AWS KMS key, also grant that identity kms:Decrypt on the key and permit it in the key policy. Finally, check the pipeline’s errors and alerts in the Dashboard.

​
Advanced

Add metadata columns to a path mapping when you need to trace destination rows to their input files. Available columns are artieSourcePath, artieSourceFileName, artieSourceLastModified, artieSourceETag, artieSourceRecordNumber, and artieSourceIngestedAt.
Artie derives the incoming row shape from parsed file data and uses normal destination schema handling. New columns can be added as they appear. When a value can be converted to an existing destination column type, Artie uses that type; incompatible values fail instead of being silently coerced. Treat renamed columns as new columns until downstream users migrate.You can change filenames without interrupting ingestion as long as each file still matches its mapping’s prefix, object filters, and optional filename expression. Use column inclusion and exclusion to prevent selected columns from being written downstream.