Breaking Apart a Monolithic ETL Codebase

I wrote about restructuring a FastAPI backend as a modular monolith — domain, application, infrastructure, presentation, each module self-contained. This is the same underlying discipline applied somewhere that discipline doesn't usually get discussed: a Databricks ETL codebase, where the default organizing principle tends to be "group by technical layer," and the result was five files that had each grown into their own small monolith.

tables_bus.py639 lines, every bus table

02_stage_to_bus.py394 lines, every bus transformation

Restructure by entity

bus/fact_replenishment/data.py + transformations.py, one folder

The Problem

A warehouse-operations reporting pipeline had grown, table by table, into a structure organized around technical layer rather than business object: one file held every bus-layer table definition, another held every bus-layer transformation, and so on for stage and serve. By the time it needed a serious touch, two files anchored the bus layer, connected only by a naming convention someone had to already know:

File	Lines	Contents
`tables_bus.py`	639	Every bus-layer table definition — 11 fact and dimension tables, back to back
`02_stage_to_bus.py`	394	Every bus-layer transformation — same 11 tables, different file, different order

The practical cost showed up in ordinary changes:

Two-file navigation for one-line changes. Adding a column to fact_replenishment meant opening the 639-line file to find its eleven-line definition block, then opening the 394-line file to find its transformation logic somewhere else in file order.
Merge conflicts between unrelated changes. Two engineers touching different tables in the same sprint collided on tables_bus.py even though their actual changes never overlapped — resolving it meant verifying that eleven unrelated table definitions hadn't been scrambled together, not just the one either engineer cared about.

Change: add a column to fact_replenishment

schematables_bus.pyfind it among 639 lines

logic02_stage_to_bus.pyfind it among 394 lines

What Grouping-By-Layer Actually Costs

Grouping by technical layer isn't an unreasonable instinct — it mirrors how the medallion architecture itself is described, raw / stage / bus / serve, and early on, when there were three tables instead of eleven, one file per layer was genuinely simpler than eleven small ones. The instinct just doesn't scale with table count, because a file organized by layer conflates two things that should be independent: how many tables exist (grows over time, unbounded) and how the codebase is organized (fixed, one file per layer, regardless of how many tables that file now holds). A monolithic table-definitions file only stays proportionate to the pipeline's actual complexity for as long as the pipeline stays small — past that point, file size becomes a function of table count, not of any actual coupling between those tables.

# tables_bus.py — before (abbreviated: 639 lines, 11 tables, one file)
class DimFacilityDetails(BaseTable):
    __tablename__ = "dim_facility_details"
    facility_sk = Column(BigInteger, primary_key=True)
    facility_id = Column(String)
    corporate = Column(String)
    # ...42 more columns

class FactReplenishment(BaseTable):
    __tablename__ = "fact_replenishment"
    replenishment_sk = Column(BigInteger, primary_key=True)
    facility_sk = Column(BigInteger, ForeignKey("dim_facility_details.facility_sk"))
    # ...18 more columns

class FactPutaway(BaseTable):
    __tablename__ = "fact_putaway"
    # ...another unrelated table, same file

The Alternatives

Split by layer, keep splitting by layer. The lighter-touch option was simply breaking the 639-line file into several smaller layer-scoped files — fact tables in one file, dimension tables in another. This helps at the margin, but it doesn't fix the underlying problem: a table's schema and its transformation logic still live in two different files, in two different directories, connected only by matching names, and every table still shares a file with several others it has no actual relationship to.

One file per table, transformations still centralized. Split the table-definitions file into one file per table, but leave the transformation script as a single per-layer orchestration file. This solves half the problem — finding a table's schema gets trivial — but the transformation logic, which is usually where the actual business rules and the actual bugs live, stays exactly as tangled as before.

Folder per entity, colocating schema and transformation. Give every table its own folder containing both its schema definition and its transformation logic, with the per-layer script reduced to orchestration — calling each entity's transformation function in the right order, not containing the logic itself.

The Decision

The third option is what shipped, and the folder structure makes the reasoning legible on its own:

warehouse_efficiency_report/
├── 02_stage_to_bus.py                  # orchestration only: import + call, in order
└── bus/
    ├── data/
    │   ├── dim_facility_details.py     # schema for this one table
    │   ├── fact_replenishment.py
    │   └── fact_putaway.py
    ├── dim_facility_details/
    │   └── transformations.py          # business logic for this one table
    ├── fact_replenishment/
    │   └── transformations.py
    └── fact_putaway/
        └── transformations.py

Touching fact_replenishment now means opening one folder — its schema and its transformation are physically next to each other, not connected by a naming convention across two large files. The orchestration script that used to hold 394 lines of transformation logic shrank to a sequence of function calls, one per entity, in dependency order:

# 02_stage_to_bus.py — after
from bus.dim_facility_details.transformations import build_dim_facility_details
from bus.fact_replenishment.transformations import build_fact_replenishment
from bus.fact_putaway.transformations import build_fact_putaway

def run(spark, cfg):
    build_dim_facility_details(spark, cfg)   # dimension first
    build_fact_replenishment(spark, cfg)     # depends on the dimension above
    build_fact_putaway(spark, cfg)

Change: add a column to fact_replenishment

bus/fact_replenishment/data.py + transformations.py, same folder

The net effect on the codebase's size was the counter-intuitive part. Splitting eleven tables and their transformations into more than a hundred small files sounds like it should add code — boilerplate imports, more file headers, more scaffolding. It didn't. The change removed more lines than it added, because a large fraction of what made the original files large wasn't unique logic — it was near-identical boilerplate repeated eleven times with minor variations, and once each table had its own small, focused file, the repetition became visible enough to actually collapse.

The Second Pass: What Splitting Made Visible

Folder-per-entity solved the navigation and merge-conflict problem, but it surfaced a different one: with each stage-layer transformation now isolated in its own small file, it became obvious that several of them contained nearly identical column-validation and date-parsing code, copy-pasted from table to table with small variations nobody had cleaned up. Nine separate staging transformations — covering source feeds from receiving, inventory location, and task-tracking tables — each had their own slightly-different version of "check this column isn't null," "normalize this header string," "parse this date format."

# Before: nearly identical logic, duplicated across nine files
# stg/facility/transformations.py
df = df.withColumn("facility_id", trim(upper(col("facility_id"))))
df = df.filter(col("facility_id").isNotNull())

# stg/task_hdr/transformations.py — same idea, drifted implementation
df = df.withColumn("task_id", upper(trim(col("task_id"))))
df = df.na.drop(subset=["task_id"])

That duplication was invisible while it lived inside two 300-plus-line files where nobody was reading the full file at once — folder-per-entity made each transformation small enough that the repetition across files became obvious by inspection. The fix was extracting the shared logic into a small, tested utilities module:

# tools/dataframe_utils/dataframe_normalizers.py
def normalize_identifier_column(df: DataFrame, column: str) -> DataFrame:
    """Trim and uppercase an identifier column, drop nulls.

    Extracted after the same trim/upper/drop-null sequence turned up,
    each time slightly differently, across nine staging transformations.
    """
    return (
        df.withColumn(column, trim(upper(col(column))))
          .filter(col(column).isNotNull())
    )

Every one of those nine call sites collapsed to a single line calling the shared function, and — because the function was now small, named, and had no Spark session or table dependencies to fake — it got a real unit test, something that wasn't practical when the same logic was buried as five inline lines inside a 100-plus-line per-table transformation function.

Splitting a monolith by entity doesn't just make each piece easier to find — it makes the duplication between pieces visible for the first time. A 639-line file hides repetition inside its own bulk; eleven 60-line files make the same repetition impossible to miss.

The Outcome

Neither change altered what the pipeline actually computed — every table produces the same output today as it did before either refactor. What changed was the shape of the codebase: