Sizing Compute as Code: Reusable Job Clusters for Databricks Pipelines

The Problem

Every job task in a family of daily ingestion pipelines pointed at the same line: existing_cluster_id: ${var.cluster_id}. One cluster, shared across every task in every job, staying warm whether or not anything was actually running on it. A slow or unhealthy task on one job could destabilize an unrelated job's run, and the cluster's size was a single compromise value — big enough for the heaviest workload that ever touched it, so every lighter job paid for capacity it didn't need. Compute wasn't scoped to the job at all; it was scoped to whatever else happened to be sharing that cluster at that moment.

Before                              After

  job A ┐                            job A → ephemeral cluster A
  job B ├─→ one shared cluster       job B → ephemeral cluster B
  job C ┘   (always warm)            job C → ephemeral cluster C
                                              (created, used, torn down)

What Job Clusters Are, and Why They're a Different Model

Databricks distinguishes between two ways a job task can get compute. existing_cluster_id references a cluster that already exists somewhere in the workspace — an all-purpose cluster someone stood up and is keeping alive, that a job just happens to attach to. job_cluster_key references a cluster defined inside the job itself, in a job_clusters block, that Databricks creates fresh when the job run starts and tears down when the run finishes. Every task in the job can reference the same job_cluster_key, so Databricks provisions one cluster for the whole job run and schedules all of that job's tasks onto it — it isn't one cluster per task, it's one ephemeral cluster shared across a single job's own task graph and nothing else:

tasks:
  - task_key: core_to_stage
    notebook_task:
      notebook_path: ${workspace.file_path}/transformations/01_core_to_stage
      source: WORKSPACE
    job_cluster_key: medium_cluster
    max_retries: 2
    min_retry_interval_millis: 300000
  # ... additional tasks, same job_cluster_key ...
job_clusters:
  - job_cluster_key: medium_cluster
    new_cluster: ${var.medium_cluster}

The cluster Databricks creates from that reference is still ephemeral — spun up at run start, torn down at run end. What's actually reusable is the specification behind ${var.medium_cluster}, not the cluster itself: Databricks Asset Bundles support a type: complex variable, effectively a whole object literal that any job can reference and get back verbatim. That's the mechanism behind "a small library of named presets every job pulls from" — three tiers, small/medium/large, each a complex variable:

variables:

  small_cluster:
    description: "Small jobs cluster — Standard_D4ds_v5, 1-2 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D4ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 1
        max_workers: 2
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: &default_spark_env_vars
        PLATFORM_ENVIRONMENT: "{env}"
        PLATFORM_TENANT_ID: "{tenant-id}"
        PACKAGE_REGISTRY_SECRET: "{{secrets/keyvault/package-registry-secret}}"
        PIP_PROXY: "{proxy-url}"
        PLATFORM_PYTHON_PACKAGE_PROXY: "--proxy {proxy-url}"
        PLATFORM_PYTHON_PACKAGE_URL: "https://token:{package-registry-secret}@{package-index-host}/simple/"

  medium_cluster:
    description: "Medium jobs cluster — Standard_D16ds_v5, 2-5 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D16ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 2
        max_workers: 5
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: *default_spark_env_vars

  large_cluster:
    description: "Large jobs cluster — Standard_E48ds_v5 (memory-optimised), 1-3 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_E48ds_v5"
      driver_node_type_id: "Standard_E48ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 1
        max_workers: 3
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: *default_spark_env_vars

Two mechanical details matter here. Complex variables replace the whole object — there's no partial merge in DAB, so a one-off tweak means overriding the entire block in that job's own new_cluster definition rather than the preset. And the spark_env_vars map is defined once, in small_cluster, using a YAML anchor (&default_spark_env_vars) aliased (*default_spark_env_vars) into the other two — a proxy URL or tenant ID change is then a one-line edit instead of three.

A job picks its tier by setting job_cluster_key to small_cluster, medium_cluster, or large_cluster — a naming decision, not a per-run tuning exercise. All three also set availability: "SPOT_WITH_FALLBACK_AZURE" with spot_bid_max_price: -1 and first_on_demand: 1: the driver always runs on-demand, workers are requested as spot at market rate, and if spot capacity isn't available or gets reclaimed mid-run, Databricks transparently falls back to on-demand instead of leaving the job stuck or failing.

Spot-with-fallback turns spot pricing from a risk you have to design around into a cost optimization you get for free — the failure mode becomes "slightly more expensive," never "the job didn't run."

Making Notebook Initialization Portable

Sizing and pricing were only half the change. The other half was the notebook every job runs first to install the platform's Python packages before any transformation logic executes. Before this work, the proxy URL used for pip installs was a literal hardcoded string sitting in the notebook itself, alongside a secret lookup that always went to the workspace secret scope regardless of whether an environment variable had already been injected by the job cluster's spark_env_vars:

# before
INDEX_URL = os.environ['PLATFORM_PYTHON_PACKAGE_URL'].format(
    package_secret=dbutils.secrets.get(scope='keyvault', key='package-registry-secret')
)
PROXY_URL = "http://{hardcoded-proxy-host}:8080"

%pip install --proxy $PROXY_URL --index-url $INDEX_URL platform-utils

That hardcoded proxy string is exactly the kind of thing the new cluster presets were designed to eliminate: with PIP_PROXY and PLATFORM_PYTHON_PACKAGE_PROXY now defined once in the shared spark_env_vars anchor and injected into every cluster the notebook could possibly run on, the notebook itself no longer needs to know what environment it's in — it just reads the variable the cluster already set:

# after
# Proxy URL for pip installations
PIP_PROXY = os.environ['PIP_PROXY']

# Secret for package registry access
PACKAGE_REGISTRY_SECRET = os.environ['PACKAGE_REGISTRY_SECRET'] or dbutils.secrets.get(scope='keyvault', key='package-registry-secret')
if PACKAGE_REGISTRY_SECRET is None:
    raise Exception("PACKAGE_REGISTRY_SECRET is not set")

INDEX_URL = os.environ['PLATFORM_PYTHON_PACKAGE_URL'].format(package_secret=PACKAGE_REGISTRY_SECRET)

# Proxy argument for pip package installations
PLATFORM_PYTHON_PACKAGE_PROXY = os.environ['PLATFORM_PYTHON_PACKAGE_PROXY']

The secret lookup keeps the secret-scope call as a fallback rather than dropping it — if the environment variable isn't set, the notebook still resolves the secret the old way, and an exception fires only if both paths come back empty. That's a deliberately conservative migration: prefer the environment-variable path the new presets provide, but don't strand a job still running on compute defined before this change.

The migration itself was scoped narrow rather than attempted everywhere at once: only the highest-volume, most cost-sensitive job family moved to job clusters in this change, with other job families left on the pre-existing shared-cluster model for the time being, that split recorded explicitly rather than left to be discovered later.

The Outcome

• Three daily pipeline definitions — roughly two dozen notebook tasks between them — moved from a shared all-purpose cluster to ephemeral job clusters, each run getting compute scoped to exactly that run, priced spot-with-fallback, and torn down when the tasks finish.
• Bundle validation confirmed the cluster configs resolved and notebooks initialized cleanly against the new proxy setup before this reached production.
• Compute sizing became something reviewable in a diff instead of an ad-hoc decision: adding a job means picking one of three named presets, not opening a conversation about node types.
• A shared config change — a proxy URL, a tenant identifier — is now a one-line edit to a single anchor instead of a hunt through every job's cluster block.
• Compute cost stopped being a property of a cluster nobody owned and became a property of the job that actually ran.