Destinations
Databricks
Learn how to configure Databricks as a destination in Artie, including SQL warehouse connection details, Unity Catalog setup, volume configuration, and authentication.
Required settings
- SQL warehouse host and path
- Unity catalog name
- Volume name
- One authentication method: personal access token or OAuth M2M service-principal credentials
Volume
Artie will write our temporary files to a volume in the catalog and schema that we are syncing into.
CREATE VOLUME my_catalog.my_schema.my_volume;Find SQL warehouse host and path
Go into your Databricks workspace, select the SQL warehouse you want to use and click on Connection details.
From there, you will be able to find the host and path of your SQL warehouse.
Create a token
In your Databricks workspace, select your username, then go to Settings > Developer > Access tokens and select Generate new token.
Add the token to Artie
In the Databricks destination configuration, select Personal access token and enter the generated token. If you set an expiration, rotate the token in Artie before it expires.
Create a Databricks service principal and generate an OAuth secret for it. Give the service principal Can use on the SQL warehouse, then grant it access to the catalog, schema, and volume by its application ID:
GRANT USE CATALOG ON CATALOG my_catalog TO `6f0c8a4e-2b1d-4c7a-9e3f-5d8b1a2c4e6f`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA my_catalog.my_schema TO `6f0c8a4e-2b1d-4c7a-9e3f-5d8b1a2c4e6f`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME my_catalog.my_schema.my_volume TO `6f0c8a4e-2b1d-4c7a-9e3f-5d8b1a2c4e6f`;If the schema already has tables Artie writes to, also grant SELECT and MODIFY on them.
In the Databricks destination configuration, select OAuth M2M (service principal) and enter the service principal’s client ID and client secret.
Don't see your schema?
Artie lists schemas through the Unity Catalog API. If a schema is missing, grant the user or service principal Artie connects as USE SCHEMA on it.
Advanced features
Turn off variant shredding
On Databricks Runtime 17.3 and later, Databricks turns on variant shredding by default for new tables with VARIANT columns. Shredding speeds up reads of VARIANT data but adds overhead to every write. For raw or bronze tables that are written often and read by only a few downstream jobs, you can turn it off to prioritize write throughput.
When you turn this setting on, Artie:
- Creates new tables, including its staging tables, with
delta.enableVariantShredding = false. - Sets
delta.enableVariantShredding = falseon the pipeline’s existing tables after each deploy. Tables that already have it turned off are skipped. - Emails the people who receive table change notifications with the list of tables it changed.
Edit your pipeline, open the destination Advanced settings, and select Turn off variant shredding under Data handling. Deploy the pipeline to apply it.
Set databricks_disable_variant_shredding on your artie_pipeline resource:
resource "artie_pipeline" "bronze" {
# ...
databricks_disable_variant_shredding = true
}What happens to data that's already shredded?
The setting only affects new writes. Data that Databricks already shredded stays shredded and can still be read normally. To rewrite it in the unshredded format, run ALTER TABLE my_catalog.my_schema.my_table DROP FEATURE variantShredding; yourself. This rewrites the table’s shredded data, so it can take a while on large tables.
Turning the setting back off doesn’t turn shredding back on for tables Artie already changed.