GMT20241122-223337_Clip_2024-11-22 — Databricks dev setup for dbt.mp4

What is this?

• This is a reference for setting up you own dbt development environment with Databricks.
• You will learn how to
  • Use VS Code (or whatever you want)
  • Install dbt the way I like to
  • Create a dev catalog in databricks
  • Wire it all up and build some models 💪

Assumptions

• I’ll be using what I like, but you can pretty painlessly swap out all of this for something you like better.
  • Don’t like VS Code? Fine, use Sublime or Emacs if you want.
  • Don’t want to develop locally? Fine, use dbt Cloud or paradime.io or prophecy.io.
  • Don’t want to use dbt at all? Hey, more power to you. Most of this should apply just fine for sqlmesh too.
  • Hell, don’t want to use databricks? Honestly a lot of this applies to any ol 3-namespace warehouse (so pretty much all of them except for Athena on Glue haha)
    • 3-namespace = database.schema.table ♥️

Warehouse Setup:

• Let’s get started by going to Databricks.
  • Don’t have an account? Well you picked a REALLY good time to get one, because Databricks now has a 100% serverless signup option.
    • Select Professional use > Express deployment (aka serverless) > provide your email and you’re done.
    • I bet you its the fastest cloud warehouse signup you’ve ever done.

Create a dev catalog (aka a dev database if you think in databases like I do)

• Ok, in Databricks we’ll go to Catalog > ➕ Add Data > Create Catalog
• My project is called randy_pitcher_workspace so I’ll call this randy_pitcher_workspace_dev
• If you aren’t sure what to call your’s, you can never go wrong with analytics_dev - its a classic for a reason : )

Create some dev compute!

• On Databricks, go to Compute > SQL > Create a SQL Warehouse

  • I like to segment my compute for dbt so I can see pricing very clearly.
  • I like to segment my production and dev warehouses to avoid spending surprises or resource contention.
  • ⚡Quick Tips!
    • Do yourself a favor and use a 2X-Small warehouse with no autoscaling unless you know for a fact you can’t
      • if you get a ton of queued queries, you have high load and should increase the scaling to handle all those queries.
      • if you get long running queries, increase the cluster size so you can rip through those queries faster
    • Add tags!
      • at least add an owner tag so folks know who to blame
      • also smart to add a use case or project tag to identify what this cluster is for
        • you’ll thank me when you have super easy usage charts by tag : )

  

VS Code Setup

• Create a repo in Github somewhere and clone it fresh 🧼
• Open VS Code and open your terminal 👩‍💻

Installing dbt