Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK

Iceberg

REST catalog

How to configure Iceberg REST Catalog as a destination in Artie

Required settings

  • REST Catalog URI
  • Authentication (Bearer token or OAuth2)
  • Catalog name

Table format

The Table format setting in the pipeline’s destination settings picks the Iceberg version for new tables. It defaults to Iceberg v2, which nearly every engine can read, including Athena, StarRocks, and older Spark and Trino versions. Tables that already have data keep their version.

Iceberg v3 stores deletes as deletion vectors, so updates and deletes write less. Pick it only when every engine that queries these tables supports v3. A table created as v3 can’t go back to v2.

Databricks Unity catalog

To configure Unity catalog as a REST catalog, you will need to do the following:

  • Enable external data access
  • Modify the service principal to have the following permissions:
    • EXTERNAL USE SCHEMA, EXTERNAL USE LOCATION
    • USE CATALOG, USE SCHEMA
    • SELECT
    • MANAGE, CREATE SCHEMA, CREATE TABLE, MODIFY, WRITE VOLUME

Grant each privilege to the service principal by its application ID. For example:

SQL
GRANT EXTERNAL USE SCHEMA ON SCHEMA catalog_name.schema_name TO `6f0c8a4e-2b1d-4c7a-9e3f-5d8b1a2c4e6f`;

Once this is done, you’ll then want to grab:

  1. REST Catalog URI (https://{{DATABRICKS_INSTANCE_URL}}/api/2.1/unity-catalog/iceberg-rest)
  2. Catalog name

Authentication

A bearer token is a personal access token, so Artie acts as the user who created it. Grant the permissions above to that user, or use OAuth2 M2M with the service principal instead.

To generate a personal access token:

Go into your workspace and do the following:

  • Click on your username in the top bar and select Settings from the dropdown.
  • Click Developer
  • Click Access tokens then Generate new token
  • (Optional) Provide a description for the token and if you set expiration, make sure to note it down.
Personal access token