We Were Supposed to Have Two Databricks Workspaces

The Naming Convention

Every target in this setup is named as <schema>-<workspace>: dev-tst, dev-prd, int-tst, int-prd, plus staging and production on their own.

The part before the hyphen — dev, int — is a Unity Catalog schema: a namespace inside a workspace, not a workspace of its own. The part after it — tst, prd — names the actual physical Databricks workspace the schema lives on. Read that way, the convention is already saying there should be two workspaces, TST and PRD, each holding both a dev and an int schema.

What Databricks Recommends

Azure Databricks' own workspace strategy guidance is explicit: "split SDLC environments into separate workspaces (at least Dev and Prod, but possibly more depending on requirements)."

Its example deployment is a production workspace, a staging workspace, and a development workspace, each with its own storage and network. Some things, like testing an admin setting before it reaches production, are only safe to do if "Prod must be a different workspace than Dev or Staging."

What This Setup Has Instead

TST was meant to be that second workspace. The deploy pipeline's own comments call tst and prd "the TST Databricks workspace" and "the PRD Databricks workspace," confirming the two halves of the naming convention really do mean what they look like they mean.

But the bundle's target comments describe TST as, in their own words, "currently not actively used — insufficient data in TST workspace." It was never provisioned. So every schema that was supposed to live on it — dev-tst, int-tst — has nowhere to go, and every stage — development, integration, staging, production — deploys into the one workspace that's actually live: PRD.

That's where it stops being cosmetic. int-prd — integration, a pre-production stage — is configured with JOBS_PERMISSION_LEVEL: CAN_MANAGE_RUN, the same production-grade access level as staging and production. It also unpauses two scheduled jobs that stay paused everywhere else, including dev-prd.

Integration testing was supposed to be a dev/test-tier concern. Instead it runs with production permissions, on a live schedule, inside the same workspace real production deploys into — because that's the only workspace there is.

Naming implies two workspacesTST + PRD, per Databricks' recommended split

TST never provisioned"currently not actively used"

dev / int / staging / productionall deploy into the one live workspace: PRD

int-prd runs prod-gradeCAN_MANAGE_RUN, jobs unpaused — a pre-prod stage with production access

The Fix: Two Paths Doing Two Different Jobs

With one workspace to work with, the fix couldn't be infrastructure — it had to be config. It rests on a distinction a bundle makes between two separate artifacts: the files it deploys, and the Terraform state it tracks them with.

databricks.yml declares both under workspace:, and only one of them is stage-aware by default:

workspace:
  root_path: ${var.deploy_path}
  file_path: ${var.deploy_path}
  state_path: /Workspace/Users/${workspace.current_user.userName}/${var.gitlab_deployment_tier}/.bundle/${bundle.name}/${bundle.target}

targets:
  TST:
    mode: production
    default: true
    presets:
      name_prefix: "[${var.gitlab_deployment_tier}] "
      trigger_pause_status: ${var.jobs_schedule_status}
      tags:
        env_tier: ${var.gitlab_deployment_tier}
        env: TST
  PRD:
    mode: production
    presets:
      name_prefix: "[${var.gitlab_deployment_tier}] "
      trigger_pause_status: ${var.jobs_schedule_status}
      tags:
        env_tier: ${var.gitlab_deployment_tier}
        env: PRD

root_path/file_path is where the deploy actually lands — the notebooks, configs, and wheel a job runs from. It's set once per stage from BUNDLE_VAR_deploy_path, the same DBX_RELEASE_PATH value each CI variable block carries (/Develop/<branch>, /Release/integration, /Release/production, and so on).

state_path, the Terraform bookkeeping that tracks what a given deploy created and owns, is built separately from ${var.gitlab_deployment_tier} and ${bundle.target} — so it always gets its own folder even on a stage where the deployed files land somewhere shared.

On top of that, every job is tagged with env_tier and env, and renamed with the [${var.gitlab_deployment_tier}] prefix. [integration] load_orders and [production] load_orders end up as two distinct, individually tagged jobs in the same workspace's job list, each deployed and torn down independently.

That's a real fix for a real problem. Without it, two stages deploying into the same workspace would either overwrite each other's Terraform state or collide on job names.

What it doesn't do, and isn't trying to do, is change what a job can access once it's running. The prefix, the tags, and the state path are how the bundle keeps two deploys from fighting over the same bookkeeping; they say nothing about compute, network, the metastore, or the permission ceiling a job runs under — which is exactly the ceiling int-prd shares with production.

The Correct Approach

Per Databricks' own guidance, the fix isn't more prefixes — it's a second workspace, reserved for production, with everything pre-production (dev, integration, staging) deployed into a separate one. That's the "at least Dev and Prod" split the guide describes, and it's the workspace this naming convention has been implying since dev-tst and dev-prd were first named apart.

A finer tst/prd distinction underneath that, if still needed, belongs as a Unity Catalog schema — the guide is explicit that narrower isolation shouldn't reach for a workspace at all.

Until that split exists, Terraform state and job-name prefixes are what keep the shared workspace from colliding. They were never meant to substitute for the isolation a second workspace gives for free.

The Outcome