GMT20240912-210035_Clip_2024-09-12 — Uniform Demo — tight as possible.mp4

What is this?

• Snowflake can read directly from the Databricks Unity Catalog
  • this uses the Databricks Iceberg REST Catalog interface for Unity Catalog
  • For your UC tables to show up as Iceberg, please enable uniform on these tables.
• This post shows you how to wire up this connectivity

What is required?

• Snowflake account and a databricks account
• You must have a Unity Catalog enabled databricks workspace
  • UC is the default for new databricks workspaces but legacy workspaces may not have them yet.
• Snowflake will need to be able to access your databricks instance
  • unless you’re a highly secure enteprise, this is almost certainly not an issue for you
  • but if you’re a fortune 100, you may have privatelink connectivity and IP restrictions to work out. But I trust you, you can do it : )

Databricks Prep

• You’ll need tables with Iceberg metadata — this is simple to do for both existing and for newly created tables.

ALTER TABLE table_name SET TBLPROPERTIES(
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg',
  'delta.columnMapping.mode' = 'name'
);

• Go ahead and generate a Personal Access Token for auth from Snowflake to Databricks. Keep it handy and we’ll come back for it
• That’s it!

AWS Prep

• Go to AWS IAM and create a role

  • Select AWS Account when prompted for what the role is for. Just select your current AWS account — we’ll be back to update this later.
  • also check the box for require external ID and just input asdf or something for now — we will be back to update this later

• Grant your role s3 read access for your bucket. Something like this is fine for the IAM Policy:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:GetObject",
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::my_bucket_name",
                  "arn:aws:s3:::my_bucket_name/*"
              ]
          }
      ]
  }
  

• We will be back to fix the trust policy 💪

• But for now, just copy the ARN of the role you just created and head to Snowflake. Keep IAM open though

Snowflake Setup:

• You’ll need 2 things before you start creating shared tables:
  1. Snowflake External Volume — this is just a way to read from S3 (or azure storage / GCP storage)
     1. storage_base_url = s3://{your_bucket_name}
     2. storage_aws_role_arn = the ARN you copied from the AWS setup
  2. Snowflake External Catalog — this is what talks to Databricks Unity Catalog. Not rocket science, but also not super well documented for what Unity Catalog expects.
     • CATALOG_NAMESPACE = {your_schema_from_databricks}
       • UC calls their top level objects Catalogs, which is a terrible name. I’d have gone with databases.
       • So for a table called my_catalog.my_schema.my_table, you’ll want CATALOG_NAMESPACE = 'my_schema'
     • CATALOG_URI = https://{your_databricks_url}/api/2.1/unity-catalog/iceberg
     • WAREHOUSE = {your_catalog_from_databricks}
       • Again, for a table called my_catalog.my_schema.my_table, you’ll want WAREHOUSE = 'my_catalog'
     • BEARER_TOKEN = your Databricks PAT from the Databricks setup step : )