Sizing Compute as Code: Reusable Job Clusters for Databricks Pipelines

The Problem

Every job task in a family of daily ingestion pipelines pointed at the same line: existing_cluster_id: ${var.cluster_id}. One cluster, shared across every task in every job, staying warm whether or not anything was actually running on it. If that cluster was unhealthy, resizing, or mid-restart from someone else's workload, every job that depended on it inherited the problem. There was no way to reason about compute cost or reliability per job because compute wasn't scoped to the job at all — it was scoped to whatever happened to be sharing that one cluster at that moment.

That coupling showed up in two distinct ways. On the reliability side, a slow or stuck task on one job could hold the shared cluster in a state that delayed or destabilized an unrelated job's run. On the cost side, the cluster's size was a single compromise value — big enough for the heaviest workload that ever touched it, which meant every lighter job was paying for capacity it didn't need, and nothing about the cluster's shape was recorded anywhere as a deliberate choice for any particular workload.

What Job Clusters Are, and Why They're a Different Model

Databricks distinguishes between two ways a job task can get compute. existing_cluster_id references a cluster that already exists somewhere in the workspace — an all-purpose cluster someone stood up and is keeping alive, that a job just happens to attach to. job_cluster_key references a cluster defined inside the job itself, in a job_clusters block, that Databricks creates fresh when the job run starts and tears down when the run finishes. Every task in the job can reference the same job_cluster_key, so Databricks provisions one cluster for the whole job run and schedules all of that job's tasks onto it — it isn't one cluster per task, it's one ephemeral cluster shared across a single job's own task graph and nothing else:

tasks:
  - task_key: core_to_stage
    notebook_task:
      notebook_path: ${workspace.file_path}/transformations/01_core_to_stage
      source: WORKSPACE
    job_cluster_key: medium_cluster
    max_retries: 2
    min_retry_interval_millis: 300000
  # ... additional tasks, same job_cluster_key ...
job_clusters:
  - job_cluster_key: medium_cluster
    new_cluster: ${var.medium_cluster}

That ${var.medium_cluster} reference is where the reusability comes from. Databricks Asset Bundles support a type: complex variable — effectively a whole object literal that any job can reference and get back verbatim. That's the mechanism that turns "one cluster spec per job" into "a small library of named presets every job pulls from":

variables:

  # SMALL — Standard_D4ds_v5 | 1-2 workers
  # Suited for: lightweight ETL, dev/test runs, low-volume incremental loads
  small_cluster:
    description: "Small jobs cluster — Standard_D4ds_v5, 1-2 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D4ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 1
        max_workers: 2
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: &default_spark_env_vars
        PLATFORM_ENVIRONMENT: "{env}"
        PLATFORM_TENANT_ID: "{tenant-id}"
        PACKAGE_REGISTRY_SECRET: "{{secrets/keyvault/package-registry-secret}}"
        PIP_PROXY: "{proxy-url}"
        PLATFORM_PYTHON_PACKAGE_PROXY: "--proxy {proxy-url}"
        PLATFORM_PYTHON_PACKAGE_URL: "https://token:{package-registry-secret}@{package-index-host}/simple/"

  # MEDIUM — Standard_D16ds_v5 | 2-5 workers
  # Suited for: standard daily/monthly ETL, most production workloads
  medium_cluster:
    description: "Medium jobs cluster — Standard_D16ds_v5, 2-5 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_D16ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 2
        max_workers: 5
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: *default_spark_env_vars

  # LARGE — Standard_E48ds_v5 (memory-optimised) | 1-3 workers
  # Suited for: heavy reprocessing, full-history loads, memory-intensive joins
  # Driver node matches worker type to avoid driver bottleneck.
  large_cluster:
    description: "Large jobs cluster — Standard_E48ds_v5 (memory-optimised), 1-3 workers"
    type: complex
    default:
      cluster_name: ""
      spark_version: "15.4.x-scala2.12"
      node_type_id: "Standard_E48ds_v5"
      driver_node_type_id: "Standard_E48ds_v5"
      enable_elastic_disk: true
      policy_id: "{cluster-policy-id}"
      data_security_mode: "USER_ISOLATION"
      runtime_engine: "STANDARD"
      autoscale:
        min_workers: 1
        max_workers: 3
      azure_attributes:
        first_on_demand: 1
        availability: "SPOT_WITH_FALLBACK_AZURE"
        spot_bid_max_price: -1
      spark_env_vars: *default_spark_env_vars

Two mechanical details matter here. First, complex variables replace the whole object — there's no partial merge in DAB. If a job needs a one-off tweak to autoscale bounds, the only option is to override the entire block in that job's own new_cluster definition rather than the preset, which is a deliberate constraint: it keeps three presets from drifting into a dozen slightly-different variants nobody can reason about. Second, the spark_env_vars map — a dozen-plus keys covering environment identity, telemetry endpoints, and package proxy configuration — is defined exactly once, in small_cluster, using a YAML anchor (&default_spark_env_vars) and aliased (*default_spark_env_vars) into the other two presets. YAML anchor scope is document-wide, so a cross-mapping reference like this works cleanly, and it means updating a proxy URL or a tenant ID is a one-line change instead of three.

The Alternatives

There were a few ways to fix the shared-cluster problem, and each had a real cost attached.

Keep the shared all-purpose cluster and just size it bigger. The simplest fix, and the one that requires no restructuring — just resize the existing cluster to cover the heaviest workload permanently. It solves nothing about reliability coupling, and it means every lighter job pays the cost of the heaviest one, all the time, whether or not the heavy job is even running that day.

Give every job its own hand-tuned cluster spec. Full control per job, and no shared-cluster reliability coupling. But with dozens of job definitions, that means dozens of independently-maintained node types, autoscale bounds, and environment variable blocks — any shared config change, like rotating a proxy URL, has to be applied everywhere by hand, and nothing stops two nearly-identical jobs from drifting into subtly different, undocumented cluster shapes over time.

Define a small number of sized presets that jobs reference by name. A middle ground: jobs get dedicated, ephemeral compute scoped to their own run, but the number of cluster shapes to reason about and maintain stays fixed and small. The cost is that a preset is a compromise for any job that doesn't fit neatly into small, medium, or large — but that's a bounded, visible trade-off rather than an unbounded one.

The Decision

The third option won, on the basis that most workloads in this pipeline family cluster into a small number of genuinely distinct shapes, and a fixed set of presets makes that visible instead of implicit. Three tiers were defined, each mapped to a workload profile rather than picked arbitrarily:

• Small — Standard_D4ds_v5, 1-2 workers. General-purpose compute, sized for lightweight ETL and low-volume incremental loads where the bottleneck is task orchestration overhead, not CPU or shuffle.
• Medium — Standard_D16ds_v5, 2-5 workers. The workhorse tier, used by the shuffle-heavy transformation stages in the daily ingestion pipelines. Standard_D16ds_v5 is a general-purpose node with enough cores and memory per executor to handle those joins without falling over on spill, while the 2-5 autoscale range absorbs the difference between a normal daily volume and a heavier catch-up run without needing a separate cluster definition.
• Large — Standard_E48ds_v5, 1-3 workers, memory-optimised. Reserved for full-history reprocessing and memory-intensive joins where the constraint is RAM per executor, not worker count — hence the narrower 1-3 autoscale range on a much larger node. The driver node type is deliberately set to match the worker type (driver_node_type_id: Standard_E48ds_v5) so the driver doesn't become the bottleneck when collecting or broadcasting large intermediate results, which is a real risk on memory-heavy jobs if the driver is left at a smaller default size.

Matching a preset to a workload is a one-time sizing decision, not a per-run tuning exercise — and that's the point of having three fixed options instead of letting every job author pick arbitrary node types and autoscale bounds. A new job doesn't require a compute-sizing conversation; it requires picking small, medium, or large based on which of the three descriptions its workload resembles.

Spot-with-Fallback: Cost Without a Reliability Trade

All three presets set availability: "SPOT_WITH_FALLBACK_AZURE" with spot_bid_max_price: -1 and first_on_demand: 1. This is Azure's spot-with-fallback behavior: the driver node (covered by first_on_demand: 1) always runs on-demand so the job doesn't lose its control node to preemption, worker nodes are requested as spot capacity at whatever the current market price is (-1 means no bid ceiling — take spot at market rate rather than risk being under-bid and starved of capacity), and if spot capacity isn't available at cluster launch or a spot worker gets reclaimed mid-run, Databricks transparently provisions on-demand capacity in its place instead of leaving the job stuck waiting or failing outright.

That combination is what makes spot viable for a production daily pipeline rather than just a dev/test cost trick: the downside of spot — the possibility of not getting capacity, or losing it mid-run — is capped by an automatic fallback, so the only thing actually at risk is paying on-demand price on the rare occasion spot isn't available, not job failure or an indefinitely queued run.

Spot pricing is a discount you get for accepting that the underlying capacity can be reclaimed. Spot-with-fallback is what turns that from a risk you have to design around into a cost optimization you get for free — the failure mode becomes "slightly more expensive," never "the job didn't run."

Making Notebook Initialization Portable

Sizing and cost were only half the change. The other half was the notebook every job runs first to install the platform's Python packages before any transformation logic executes. Before this work, the proxy URL used for pip installs was a literal hardcoded string sitting in the notebook itself, alongside a secret lookup that always went to the workspace secret scope regardless of whether an environment variable had already been injected by the job cluster's spark_env_vars:

# before
INDEX_URL = os.environ['PLATFORM_PYTHON_PACKAGE_URL'].format(
    package_secret=dbutils.secrets.get(scope='keyvault', key='package-registry-secret')
)
PROXY_URL = "http://{hardcoded-proxy-host}:8080"

%pip install --proxy $PROXY_URL --index-url $INDEX_URL platform-utils

That hardcoded proxy string is exactly the kind of thing the new cluster presets were designed to eliminate: with PIP_PROXY and PLATFORM_PYTHON_PACKAGE_PROXY now defined once in the shared spark_env_vars anchor and injected into every cluster the notebook could possibly run on, the notebook itself no longer needs to know what environment it's in — it just reads the variable the cluster already set:

# after
# Proxy URL for pip installations
PIP_PROXY = os.environ['PIP_PROXY']

# Secret for package registry access
PACKAGE_REGISTRY_SECRET = os.environ['PACKAGE_REGISTRY_SECRET'] or dbutils.secrets.get(scope='keyvault', key='package-registry-secret')
if PACKAGE_REGISTRY_SECRET is None:
    raise Exception("PACKAGE_REGISTRY_SECRET is not set")

INDEX_URL = os.environ['PLATFORM_PYTHON_PACKAGE_URL'].format(package_secret=PACKAGE_REGISTRY_SECRET)

# Proxy argument for pip package installations
PLATFORM_PYTHON_PACKAGE_PROXY = os.environ['PLATFORM_PYTHON_PACKAGE_PROXY']

The secret lookup keeps the secret-scope call as a fallback rather than dropping it — if the environment variable isn't set for some reason, the notebook still resolves the secret the old way instead of failing outright, and an explicit exception fires only if both paths come back empty. That's a deliberately conservative migration: prefer the environment-variable path the new cluster presets provide, but don't strand a job that runs on compute defined before this change rolled out everywhere. The rest of the notebook was also broken into named, labeled steps instead of one long unlabeled block — not a functional change to what installs, but it matters for the portability goal: when the notebook runs against a new environment for the first time and something in the install chain fails, it's now visible which named step failed instead of a wall of output with no obvious boundary between fetching the index and installing the requirements.

The migration itself was scoped deliberately narrow rather than attempted everywhere at once. Only the highest-volume, most cost-sensitive job family moved to job clusters in this change; other job families kept using the pre-existing shared-cluster model for the time being, with that split recorded explicitly rather than left to be discovered later. Migrating every job family in one pass would have meant a much larger blast radius for a change whose main value — decoupling reliability and making cost visible — was already captured by moving the biggest workload first.

The Outcome

Three daily pipeline definitions — spanning roughly two dozen notebook tasks between them — moved from a shared all-purpose cluster to ephemeral medium_cluster job clusters. Each job run now gets a cluster scoped to exactly that run, provisioned with spot-with-fallback pricing, and torn down when the tasks finish, instead of a shared cluster that stayed warm and billable independent of whether anything was using it. Bundle validation confirmed the cluster configurations resolved correctly and notebooks initialized cleanly against the new environment-variable-driven proxy setup before this reached production, catching any misconfiguration at deploy time rather than at the first job run.

The bigger, less immediately visible result is structural: compute sizing became something reviewable in a diff instead of an ad-hoc decision made once and never revisited. Adding a new job now means picking one of three named presets rather than opening a conversation about node types and autoscale bounds, and a shared config change — a proxy URL, a tenant identifier — is a one-line edit to a single anchor instead of a hunt through every job's cluster block to make sure none of them drifted.