Databricks still has no native reader for .xlsb. .xlsx and legacy .xls are covered by community libraries and pandas' openpyxl engine, but the project's source exports were binary .xlsb workbooks specifically — the one Excel format nothing on the platform reads out of the box. That gap is what this framework exists to close.
Config-Driven, Not Notebook-Driven
Every source is a row in a config table, not a notebook: its file path, target table, load type (append/overwrite), and a source_type field — excel, csv, or delta_table. Onboarding a new source is a config change, not a new notebook copy-pasted from the last one. A FileHandlerFactory reads source_type and routes to the matching handler class; adding a new source type is a new handler and one line in the factory, not another branch in a growing conditional.
Making .xlsb Work
The Excel handler resolves its pandas engine from the file extension, not from a config flag: .xlsb always routes to pyxlsb, everything else falls through to pandas' openpyxl default.
class ExcelEngineResolver:
@staticmethod
def resolve(file_path: str) -> Optional[Literal["pyxlsb"]]:
""".xlsb files ALWAYS use pyxlsb. .xlsx / .xls use the pandas default."""
if file_path.lower().endswith(".xlsb"):
return "pyxlsb"
return None
Sheet discovery has to branch the same way — pyxlsb.open_workbook for .xlsb, pd.ExcelFile for everything else — because pyxlsb is the only engine that reads sheet names out of the binary format reliably.
Past that split, the rest of the pipeline doesn't care which engine loaded the sheet: pandas hands back a DataFrame, it's converted to Spark, columns are validated and normalized, and every column is cast to string before the write. Type drift between an .xlsx export and an .xlsb export of the same report can't silently produce two different schemas.
Scanning the Volume
Before any file is matched to a config, it has to be discovered. Each run calls dbutils.fs.ls against the storage path and turns every entry into a FileInfo — path, name, size, and a modification_timestamp read straight off modificationTime. That timestamp is the one piece of metadata everything downstream keys on.
Duplicate-Load Prevention
The framework doesn't ingest the same file twice for the same version of it. Every successful load writes an audit row keyed on source path, source name, and that same modification timestamp from dbutils.fs.ls.
Before touching a file, the handler checks whether a SUCCESS row already exists with a source_modified_timestamp at or after the one on disk:
SELECT count(1) AS num
FROM log_file_load_control
WHERE source_path = '{source_path}'
AND source_name = '{file_name}'
AND source_modified_timestamp >= '{source_timestamp}'
AND status = 'SUCCESS'
A file that's already loaded is skipped. A file that's been replaced with a newer modification timestamp reads as new and gets reprocessed.
Nothing about this is exactly-once — a file rewritten in place with an unchanged timestamp would still read as already loaded — but it's enough to stop the ordinary case of a scheduled job re-scanning a folder that hasn't changed since the last run.
File names aren't always literal either. Several sources land with a date stamp baked into the name — a monthly export like report_202609.xlsb — so configs can declare a <yyyyMM> placeholder instead of one exact file name, matched with a regex built from the config at run time:
pattern = re.escape(config.file_nm).replace(re.escape(""), r"\d{6}")
pattern = r"^" + pattern
return re.match(pattern, file_name) is not None
One config row then matches every month's file without being rewritten each month. Any file in the volume that matches no config — placeholder or exact — gets logged as MISSING_CONFIG instead of silently ignored.
CSV: The Slow Move to Autoloader
CSV support was added to the same factory as a third handler, deliberately reusing the framework's existing shape rather than introducing a second ingestion pattern: a config row, a handler, the same DataWriter and audit log as Excel.
Today that handler still reads with Spark's plain batch CSV reader — read_parameters on the config are passed straight through as reader options, with header and inferSchema always enforced regardless of what a config sets, so schema inference can't silently vary source to source.
That batch reader is the piece actually being phased out. It works, but it re-lists and re-evaluates an entire volume on every run — via the same dbutils.fs.ls scan as Excel — instead of tracking incrementally what it has already seen. Fine at the current file counts, but it's the one part of the framework not built around incremental discovery.
The direction is Databricks Autoloader for CSV specifically: Autoloader tracks which files it has already seen internally, so the same skip-if-already-loaded job the audit log does by hand for Excel is closer to a platform primitive for CSV. Excel stays on the pandas/engine-resolver path — Autoloader has no .xlsb story either — so the migration is scoped to the one source type where a genuinely better option already exists, not a rewrite of the whole framework at once.
Outcome
.xlsbingestion works today because the pandas engine is resolved per file extension instead of assumed — the one branch that made the rest of the framework usable for the files the project actually receives.- Adding CSV and Delta-to-Delta handlers later didn't touch the factory, the config model shape, or the audit log — each was an isolated addition behind the same routing.
- The audit log's timestamp check, not a content hash, is what keeps a scheduled rerun from reloading a file that hasn't changed — cheap, and precise enough for how these sources actually get updated.
- CSV is moving to Autoloader incrementally, one source type at a time, instead of a framework-wide rewrite — Excel keeps its pandas path because Autoloader doesn't solve the
.xlsbproblem this framework was built to solve.