Dependency Vulnerabilities as a Pipeline, Not a One-Off Fire Drill

The Problem

For a long stretch, dependency vulnerabilities were handled the same way every time: somebody — a teammate, a security bulletin, occasionally a CI failure from an unrelated cause that happened to surface an old warning — noticed that a pinned library version had a disclosed CVE, and the fix was a one-line commit pinning a patched version. It worked, in the sense that every individual CVE eventually got patched. It didn't scale, in the sense that "eventually" depended entirely on someone happening to notice, and noticing wasn't anybody's actual job.

The pattern showed up constantly across a single dependency tree: cryptography, aiohttp, starlette, idna, pyjwt, and an internal shared toolkit library all needed at least one reactive security pin over the life of the project — each one a separate, manually-triggered fire drill, each one only as fast as the gap between disclosure and someone reading about it.

What Actually Changed

The scanning itself wasn't something this project built. The platform/infra team maintains a shared GitLab CI/CD component — a reusable pipeline definition, versioned and included by reference rather than copy-pasted — that resolves a repo's requirements.txt against a private package index the same team also runs. The engineering work on this side wasn't writing a scanner; it was integrating that component correctly, understanding the private index's quarantine policy well enough to react to it sensibly, and building somewhere for its findings to actually land.

1. Pulling in a shared component instead of building one

GitLab's CI/CD components let one team publish a reusable, versioned pipeline definition that other repositories include by reference, the same way a library gets imported rather than copy-pasted. Including the platform team's component meant every merge request in this repo now resolved its full dependency tree — direct and transitive — against their private index as part of CI, without this project maintaining any scanning logic of its own:

# .gitlab-ci.yml (excerpt)
include:
  - component: $CI_SERVER_FQDN/platform/cicd-components/dependency-scan@2.3.0

dependency-scan:
  extends: .dependency-scan-component
  variables:
    REQUIREMENTS_FILE: requirements.txt

That's a build-versus-adopt decision as much as a technical one. Building an equivalent scanner from scratch would have meant maintaining a CVE feed, a resolver, and a severity model independently — duplicating work the platform team was already doing centrally, and doing it worse, since they run the index those CVEs actually get enforced against. Adopting their component meant this repo's scanning behavior stayed in step with every other repo using the same component, for free, every time the platform team shipped an update to it.

2. Understanding why "pip install" started failing

The private index doesn't just report vulnerabilities — it enforces against them. Any quarantined version produces the same visible symptom: pip install --index-url <private-index> returns an HTTP 403 for that specific version, not a warning, not a slow resolution, a hard failure a build has to react to. What triggers that 403 is one of two independent policies:

The first row is a blanket rule, unrelated to whether anything is actually wrong with a given release yet. It exists for a specific supply-chain attack shape:

• A maintainer account gets compromised.
• A malicious release goes out under a trusted package name.
• Every consumer who upgrades within the first few hours or days pulls it in before anyone's noticed anything is wrong.

A seven-day hold gives the wider ecosystem — security researchers, other consumers, the package's own maintainers — time to catch a bad release before this index will serve it to anyone at all.

The second row is different in kind: it isn't about a release being new, it's a forced deadline on a version already sitting in the index once a CVE is filed against it. Fourteen days is a grace period, not a suggestion — a scheduled job on the platform team's side enforces it automatically once the clock runs out, whether or not anyone has acted.

The two produce an identical 403, but they call for opposite responses:

• New-release quarantine — wait it out. Nothing is actually wrong; the version resolves normally once the seven days pass.
• CVE quarantine — treat it as a countdown. The patched version needs adopting before day fourteen, or the pinned one stops resolving with no further warning.

Telling those two apart is why an alert needs to say why a package was flagged, not just that it was.

3. Turning a job log nobody read into thirteen days of runway

The component already printed every vulnerable package it found straight to the job's stdout on each run — the information needed to act early was there from day one, sitting in a CI log nobody was in the habit of opening unless the job had gone red. It hadn't gone red yet for a CVE still inside its fourteen-day grace period; the scan reported the finding and passed. That's exactly the failure mode the manual-noticing era had: the signal existed, but only for someone who went looking for it, in a log for a job that technically succeeded.

The scheduled scan runs daily, independent of whether anyone opens a merge request, so a newly-disclosed CVE against something already pinned typically surfaces in that stdout output within a day of the disclosure itself — not at day fourteen when quarantine was about to hit. That timing is what made the second half of this worth building: parsing the same stdout the job was already producing and forwarding it to a Microsoft Teams channel, created specifically for this, via an incoming Workflows webhook.

def notify_quarantine_risk(package: str, version: str, cve_id: str, disclosed_on: str, quarantine_date: str) -> None:
    """Post a quarantine warning to the dedicated Teams channel via a Workflows webhook.

    Parses the same stdout the scheduled scan already printed to the CI
    job log. The gap this closes isn't detection — the scanner already
    detected it and the log already had it — it's making the finding
    visible the day it happens instead of the day someone happens to
    read that job's log.
    """
    days_remaining = (parse_date(quarantine_date) - today()).days
    payload = {
        "text": (
            f"`{package}=={version}` has {cve_id} (disclosed {disclosed_on}) "
            f"and will be quarantined on {quarantine_date} — "
            f"{days_remaining} days to patch, rebuild, and test before "
            f"the pinned version stops resolving with no further warning."
        )
    }
    requests.post(TEAMS_WORKFLOW_WEBHOOK_URL, json=payload, timeout=10)

In practice, catching a disclosure the day after it happens instead of the day quarantine enforces it turned a fourteen-day grace period into roughly thirteen usable working days — enough time to bump the pinned version, rebuild, and run the test suite against it deliberately, before the deadline forced the issue by making the old version stop resolving. Without that runway, the same fourteen-day clock still ran, but nobody was watching it start; the first signal anyone actually acted on was the 403 itself, on day fourteen, with the CI/CD pipeline already blocked and, since the same private index also serves library installs for the Databricks job clusters, the production data pipelines blocked right along with it. A dedicated channel — no other traffic in it — was a deliberate choice on top of that: a general team channel buries a security alert under everything else within a day, while a channel that only ever contains quarantine warnings makes an empty channel the normal state and any message in it worth opening immediately.

4. Knowing when automation should still defer to a human

One incident made clear that even a well-understood quarantine policy doesn't remove the need for judgment. An internal shared toolkit library was bumped from an older pinned version to a newer one specifically to get ahead of its own upcoming quarantine deadline, and the newer version introduced an unrelated regression serious enough to need reverting the bump entirely — back to the version nearing quarantine — while the actual fix was worked around separately in the meantime. An automated "always take whatever version isn't quarantined yet" policy would have shipped that regression straight through, on schedule, exactly when the deadline said to.

The quarantine policy's job is making sure nobody has to notice a CVE by luck, and making a malicious release unusable before most consumers ever see it. It was never meant to make the "is this specific patched version actually safe to ship" call — that part stayed a human decision on purpose, and the one time it would have been convenient to skip that step is exactly when skipping it would have shipped a regression.

The Alternatives Considered

Build an independent scanner instead of adopting the shared component. Full control over severity thresholds and alerting logic, at the cost of duplicating a CVE feed and a resolver the platform team already runs centrally — and drifting out of sync with the index's own quarantine behavior, which no independently-built scanner would know about unless it specifically modeled the same 7-day and 14-day rules.

Rely on the component's CI gate alone, skip the dedicated alert channel. Simpler — no webhook, no extra channel to maintain — but it leaves exactly the gap described above: a dependency already deployed, sitting untouched while its fourteen-day quarantine clock runs out, gets no signal at all until the next unrelated merge request happens to trigger a fresh scan.

Auto-merge any dependency bump that clears the scanner. Would have closed the loop fastest for the common case. The toolkit-library regression is the argument against it: the one time a version that "cleared quarantine" wasn't actually safe to ship, the cost of that single incident outweighed the cumulative time saved by skipping review on every other one.

The Outcome

Vulnerability handling moved from "whoever notices, whenever they notice" to a pipeline built mostly out of someone else's platform investment, integrated correctly: the shared component's CI gate stops a newly-vulnerable pin from being introduced, its scheduled scan was already printing every new finding to a job log the day it appeared, and the Teams webhook is what turned that unread stdout into roughly thirteen days of working runway instead of a fourteen-day clock nobody was watching. The index's blanket seven-day quarantine on every new release closes the supply-chain window this project never had to design for itself. In practice, that runway is what usually keeps a CVE from ever becoming an incident at all: the version gets bumped, rebuilt, and tested well before day fourteen, so quarantine never gets the chance to block a merge request or take down the Databricks job clusters that pull from the same index. A human still decides whether a specific patched version is safe to actually ship — the toolkit-library regression is why that step stayed manual on purpose. What the pipeline replaced wasn't judgment. It was discovery by luck, which never needed a human in the loop to begin with.