Automating Spatial Privacy Checks in CI

Every control on this site fails the same way in production: it is applied correctly once, documented in a design note, and then silently stops applying when a schema changes, a resolution is tuned, or a new export is added. A privacy check that runs in CI and blocks the pipeline is the only version of a control that survives contact with a team.

What Belongs in CI and What Does Not # permalink

A useful split is between invariants and judgements. An invariant is a property of the output that must hold for every release and can be evaluated mechanically: no cell below the k floor, no coordinate with more than four decimal places, the CRS is projected before the noise call, the cumulative ε is under the ceiling. A judgement is a decision about the threat model: whether the audience is public, whether an auxiliary layer is plausible, whether an attribute is sensitive.

Invariants belong in CI. Judgements belong in the release record, where they are reviewed by people — and where they become inputs that CI reads. This is the split that makes the automation tractable: the pipeline does not decide whether $k$ should be 5 or 11, it enforces whatever the record says and fails if the record is missing.

The consequence is that the checks are cheap. Almost every one is a few lines over a dataframe, and their value comes not from sophistication but from running on every commit and blocking the merge.

The Check Inventory # permalink

Structural invariants # permalink

These hold regardless of the technique chosen and are the cheapest to add.

Guarantee invariants # permalink

These enforce the specific claim the release makes.

where $U(c)$ counts distinct individuals in cell $c$ — never rows. A k check over row counts is the single most common false pass, for the reasons set out in k-anonymity grouping for location traces.

Negative controls # permalink

A check that has never rejected anything proves nothing. Every suite should feed itself input it must refuse: a frame with one person in a cell, a geographic CRS, an ε that exceeds the ceiling, a coordinate at seven decimal places. If the negative control passes, the check is not wired up — and this failure is invisible without the control, because a green suite looks identical either way.

Prerequisites & Data Requirements # permalink

• A machine-readable release record. The thresholds have to live somewhere CI can read: a YAML or JSON file under version control carrying $k$, $\varepsilon_{\max}$, $\rho_{\max}$, the declared CRS, the precision, and the reviewer. This is the same artefact described in building a spatial privacy audit report template.
• A representative fixture that is not the production data. CI needs a dataset with the same shape — a rural tail, repeat visitors, a boundary-straddling cell — that can live in the repository. Synthesising it is a legitimate use of synthetic mobility data generation.
• A budget ledger with durable storage. Cumulative ε cannot be recomputed from the current run; it has to be read from and written to a store that survives the job.
• Pinned dependencies. A check whose verdict depends on a floating GEOS or PROJ version is not reproducible, for the reasons in pinning geospatial dependencies for reproducible releases.

Step-by-Step Implementation # permalink

Step 1 — Put the thresholds in version control # permalink

# release-policy.yml — the only place a threshold is written down
dataset: mobility_od_monthly
audience: public
k_min: 10
epsilon_ceiling: 4.0
reid_ceiling: 0.05
crs: "EPSG:32633"
coordinate_decimals: 4
identifier_allowlist: [cell_id, period, count]
reviewer: "privacy-office"

A threshold that appears only in a function default is a threshold nobody reviews. Reading it from a file makes every change a diff.

Step 2 — Assert the structural invariants over the built artefact # permalink

import geopandas as gpd

def assert_structure(gdf: gpd.GeoDataFrame, policy: dict) -> None:
    """Cheap invariants that hold for every release, whatever the technique."""
    assert gdf.crs is not None and gdf.crs.is_projected, "output CRS is not projected"
    assert gdf.crs.to_string() == policy["crs"], "output CRS is not the declared one"
    extra = set(gdf.columns) - set(policy["identifier_allowlist"]) - {"geometry"}
    assert not extra, f"unexpected columns in the release: {sorted(extra)}"
    assert gdf.geometry.is_valid.all(), "invalid geometries in the release"

Step 3 — Assert the guarantee on distinct individuals # permalink

import pandas as pd

def assert_k_floor(records: pd.DataFrame, k_min: int) -> None:
    """Count people, never rows. A frequent visitor must not lift a cell over k."""
    per_cell = records.groupby("cell_id")["person_id"].nunique()
    worst = int(per_cell.min())
    assert worst >= k_min, (
        f"cell {per_cell.idxmin()} holds {worst} distinct individuals, "
        f"below the declared floor of {k_min}"
    )

Step 4 — Refuse to publish over the budget ceiling # permalink

def assert_budget(ledger: pd.DataFrame, this_release_eps: float,
                  ceiling: float, dataset: str) -> float:
    """Sequential composition over every prior release from the same source."""
    spent = float(ledger.loc[ledger["dataset"] == dataset, "epsilon"].sum())
    total = spent + this_release_eps
    assert total <= ceiling, (
        f"{dataset}: {spent:.2f} already spent, this release adds "
        f"{this_release_eps:.2f}, ceiling is {ceiling:.2f}"
    )
    return total

Step 5 — Feed the suite something it must reject # permalink

import pytest

def test_k_floor_rejects_a_thin_cell():
    """Negative control: if this passes, the gate is not wired to the pipeline."""
    thin = pd.DataFrame({"cell_id": ["a"] * 3, "person_id": [1, 1, 2]})
    with pytest.raises(AssertionError):
        assert_k_floor(thin, k_min=10)

def test_structure_rejects_a_geographic_crs():
    gdf = gpd.GeoDataFrame(geometry=gpd.points_from_xy([0.0], [0.0]), crs="EPSG:4326")
    with pytest.raises(AssertionError):
        assert_structure(gdf, {"crs": "EPSG:32633", "identifier_allowlist": []})

Validation & Re-identification Testing # permalink

The dataset assertions above are necessary and not sufficient: they check the thresholds, not the risk. A scheduled job should re-run the full re-identification risk assessment against the current auxiliary landscape and open a ticket when the estimate crosses the ceiling — because that estimate moves without the release changing, as new public layers appear.

Report the risk figure per zone rather than pooled. A pooled estimate is dominated by the dense majority and hides the periphery, and the periphery is where the failures are.

Keep the suite’s own history. A check that started failing three releases ago and was skipped is worse than one that was never written, because the release record will claim it ran.

Common Failure Modes & Gotchas # permalink

Checks that run after publication. A nightly report that finds a k violation documents a breach. Move it into the publish job.

Row counts standing in for person counts. The most common false pass, and invisible until someone looks at a specific cell.

Fixtures that are too clean. A fixture without a sparse tail exercises none of the paths that fail in production. Build the fixture to contain the awkward cases deliberately.

Thresholds that live in defaults. A default parameter is a threshold nobody diffed. Read them from the policy file and fail loudly if it is missing.

Skipping the negative controls. Without them a disconnected gate and a passing gate are indistinguishable.

Treating the budget as recomputable. Cumulative ε is a property of everything ever published from that source, so it has to be stored, not derived.

What a Failing Check Should Do # permalink

A gate is only as useful as what happens when it fires, and three responses are common. Only one of them is right.

Blocking the job is the correct default. The release does not publish, the build is red, and somebody has to change either the data or the policy. That friction is the point: it converts a privacy decision from something a person can forget into something a person has to make.

Warning and continuing is the response teams reach for when a gate is new and they are not yet confident in it. It is defensible for exactly one release cycle, as a calibration run, and it should carry an expiry. A warning that has been firing for six months is a check that has been disabled without anybody deciding to disable it, and the release record still says the check ran.

Skipping under a flag is how a suite dies. An override that exists is an override that will be used under deadline pressure, and its use leaves no trace unless the flag is itself logged and reviewed. If a legitimate release genuinely cannot pass, the honest mechanism is a change to the policy file — a diff, a reviewer, a record — rather than an argument passed to the job.

There is a fourth response worth designing for explicitly: failing with a counter-proposal. A gate that says “k floor not met” leaves the operator to work out what to do. A gate that says “k floor not met at 250 m; the same policy is satisfiable at 500 m with 14% suppression” turns a rejection into a decision. Every check in this suite can compute that alternative cheaply, because it already has the sweep, and doing so is the difference between a gate people route around and one they use.

Keeping the Suite Honest Over Time # permalink

Three habits matter more than any individual check.

Add a fixture row for every production bug. The fixture becomes a record of what has actually gone wrong, and it grows in the direction of real failure modes rather than imagined ones.

Re-run the negative controls on every commit, in the same job. Splitting them into a separate workflow means a partial run can report green while the controls never executed.

Review the policy file’s history, not just its contents. A threshold that has been lowered twice in a year is telling you something the current value cannot.

Starting From Nothing # permalink

A team with no automated checks does not need the full suite on day one, and trying to build it at once is how the effort stalls. Three checks, in this order, deliver most of the value.

First, the CRS assertion. It is one line, it never produces a false positive, and it catches the single most damaging class of error on this site — a noise scale or a threshold applied in degrees. Adding it costs an afternoon and removes an entire failure mode.

Second, the k floor over distinct individuals. It is the guarantee most releases actually claim, and the row-count version of it is the most common way that claim is false. Wire it into the publish job, not a report.

Third, the negative controls for both. Without them the first two are indistinguishable from checks that have quietly stopped running, and a suite that cannot detect its own disconnection will eventually be one.

Everything else — budget ledgers, suppression completeness, scheduled risk re-simulation — is worth adding, and none of it compensates for the absence of those three.

Compliance Alignment # permalink

The strongest compliance argument this produces is not the checks themselves but their history: a regulator can see that the k floor has been enforced on every release since a specific commit, and can see the commit that changed it.

FAQ # permalink

Should CI have access to production data?

Preferably not. Run the structural and unit checks on a fixture in CI, and run the dataset assertions inside the release job where the data already lives. The important property is that the assertions block the publish, not that they run on a particular machine.

How do I check ε when the release is not differentially private?

You do not — you check the technique’s own invariant instead. A k-anonymity release checks the minimum class size and the suppression completeness; a masked release checks the displacement distribution and the resampled fraction. The pattern is the same: whatever the release record claims, the job asserts.

What if a check fails on a legitimate release?

That is the check working. The correct response is to change the release — coarsen, suppress, or spend more budget — or to change the policy file with a review. Adding a skip is how the suite stops meaning anything.

Is this worth it for a single annual release?

Yes, and more so: a control applied once a year is a control nobody remembers the details of. The suite is the memory.

Related # permalink

• Writing pytest Assertions for k-Anonymity Guarantees — the test module in full, including its negative controls
• Gating Releases on a Re-identification Risk Budget — turning a risk estimate into a blocking condition
• Building a Spatial Privacy Audit Report Template — the record these checks read from and write to
• Re-identification Risk Assessment for Geospatial Datasets — the measurement the scheduled job re-runs
• Python Spatial Privacy Toolkit — the libraries these checks are written against

← Back to Spatial Privacy Fundamentals & Threat Modeling