Pinning Geospatial Dependencies for Reproducible Releases

A privacy release is an assertion — every cell holds at least $k$ people, every point stayed inside its district — and an assertion is only as reproducible as the library that evaluated it. In the Python spatial stack that library is usually C, it is usually not what requirements.txt names, and a minor version bump can move a point across a boundary.

Core Specification # permalink

The Python packages are thin wrappers. What actually decides a contains or a coordinate transform is the native stack beneath them:

The property a reproducible release needs is that a rerun produces byte-identical output:

for environments $E_1, E_2$ built from the same lock. This fails when any of four things is unpinned: the Python packages, the native libraries, the datum grids, or the random seed. Pinning only the first is the common state, and it is the one that produces a release nobody can reproduce six months later.

The environment fingerprint is the object that makes the failure visible:

Record it with every release. When two runs disagree, the fingerprint says whether the environment changed before anyone starts bisecting the code.

Worked numeric example # permalink

The same masking pipeline, same input, same seed, run in three environments:

Env A and Env B have identical requirements.txt and produce different releases. 214 points moved district; three cells fell below $k = 10$ and were suppressed in B but published in A. If A was the run that was reviewed and B was the run that shipped, the release published three cells the review had approved and the pipeline had, in a different environment, refused.

Env C differs from B in the Python wrapper only, and produces an identical output — confirming the wrapper version was never the variable.

Python Implementation # permalink

Capture the whole stack, not the wheels:

from __future__ import annotations

import hashlib
import json
import platform
import sys


def environment_fingerprint() -> dict:
    """Record every layer that can change a geometric predicate.

    Python package versions alone are insufficient: shapely 2.0.4 links
    whatever GEOS the wheel or the system supplied, and a GEOS minor bump
    changes `contains` on boundary-coincident geometry.
    """
    parts: dict[str, str] = {
        "python": sys.version.split()[0],
        "platform": platform.platform(),
    }

    try:
        import shapely
        parts["shapely"] = shapely.__version__
        parts["geos"] = shapely.geos_version_string
    except ImportError:
        pass

    try:
        import pyproj
        parts["pyproj"] = pyproj.__version__
        parts["proj"] = pyproj.proj_version_str
        # Datum grids live outside the wheel and change transform results by
        # metres. Their presence is part of the environment, not of the code.
        parts["proj_data_dirs"] = ";".join(sorted(pyproj.datadir.get_data_dir().split(";")))
    except ImportError:
        pass

    try:
        from osgeo import gdal
        parts["gdal"] = gdal.__version__
    except ImportError:
        pass

    try:
        import numpy
        parts["numpy"] = numpy.__version__
        parts["blas"] = str(numpy.__config__.get_info("blas_opt_info").get("libraries", []))
    except Exception:
        pass

    blob = json.dumps(parts, sort_keys=True).encode()
    return {"components": parts, "fingerprint": hashlib.sha256(blob).hexdigest()[:16]}


def assert_environment(expected_fingerprint: str) -> None:
    """Refuse to produce a release from an unrecorded environment."""
    actual = environment_fingerprint()
    if actual["fingerprint"] != expected_fingerprint:
        raise RuntimeError(
            f"environment fingerprint {actual['fingerprint']} does not match the "
            f"pinned {expected_fingerprint}; components: "
            f"{json.dumps(actual['components'], indent=2)}")

Pin the native stack with a conda-lock or a container digest, not a version range:

# environment.yml — versions here pin the native libraries, not just wheels
name: spatial-privacy-release
channels: [conda-forge]
dependencies:
  - python=3.12.4
  - geos=3.12.1
  - proj=9.4.0
  - proj-data=1.19          # the datum grids, which change coordinates by metres
  - gdal=3.9.1
  - shapely=2.0.6
  - pyproj=3.6.1
  - geopandas=1.0.1
  - numpy=1.26.4

# Reference the image by digest. A tag is a moving target and "3.12-slim"
# has silently changed its GEOS more than once.
FROM condaforge/miniforge3@sha256:9f2c1d3e...
COPY environment.yml /tmp/
RUN conda env create -f /tmp/environment.yml && conda clean -afy

Verification # permalink

Two clean builds must agree byte for byte. Build the environment twice from the lock, run the pipeline, compare hashes:

def reproducibility_check(run_a: bytes, run_b: bytes,
                          fp_a: str, fp_b: str) -> dict:
    """Distinguish an environment drift from a genuine non-determinism bug."""
    return {
        "outputs_match": hashlib.sha256(run_a).hexdigest() ==
                         hashlib.sha256(run_b).hexdigest(),
        "environments_match": fp_a == fp_b,
        # Same environment, different output means unseeded randomness,
        # dict ordering, or a timestamp in the artefact — a code bug.
        # Different environment, different output means the lock is leaky.
        "diagnosis": ("code non-determinism" if fp_a == fp_b else "leaky lock"),
    }

Assert the boundary predicates directly. A geometry fixture of points sitting exactly on boundaries is the cheapest early warning of a GEOS change, and it fails at import time rather than at release time:

BOUNDARY_FIXTURES = [
    # (point, polygon_wkt, expected_contains) — chosen to sit on the edge,
    # which is where GEOS versions disagree and interior points never do.
    ((0.0, 0.0), "POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))", False),
    ((0.5, 0.0), "POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))", False),
    ((0.5, 0.5), "POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))", True),
]

def test_geos_predicates_unchanged():
    from shapely import Point, from_wkt
    for (x, y), wkt, expected in BOUNDARY_FIXTURES:
        assert from_wkt(wkt).contains(Point(x, y)) is expected

Assert the transform to the metre. Pick three control points with known projected coordinates and assert them to 1 mm. A datum-grid change shows up here as a systematic few-metre shift, which is exactly the magnitude that silently reassigns points near boundaries:

CONTROL_POINTS = [
    # (lon, lat, expected_easting, expected_northing) in the release CRS
    (-0.001545, 51.477928, 539_985.19, 177_297.34),
]

def test_projection_unchanged(tol_m: float = 0.001):
    from pyproj import Transformer
    t = Transformer.from_crs("EPSG:4326", "EPSG:27700", always_xy=True)
    for lon, lat, e, n in CONTROL_POINTS:
        got_e, got_n = t.transform(lon, lat)
        assert abs(got_e - e) < tol_m and abs(got_n - n) < tol_m

Record the fingerprint in the release artefact itself. Not in a log — in the file. A release whose environment cannot be identified from the release is not reproducible in any useful sense, because the log will have rotated by the time anyone asks.

Edge Cases & Adjustments # permalink

• Wheels bundle their own natives. A pip install shapely on Linux ships a vendored GEOS inside the wheel, so two machines with identical pip freeze output can still differ if one built from source. shapely.geos_version_string is the ground truth; the package version is not.
• proj-data is optional and changes results. Without the grid files PROJ falls back to a coarser datum transformation, differing from the grid-based answer by up to several metres. Its presence must be pinned, not merely permitted.
• Floating-point and BLAS. Distance computations through scipy.spatial can differ in the last bits across BLAS implementations. That rarely matters — until a comparison is d <= r_max and a point sits exactly at $r_{\max}$. Compare with an explicit tolerance rather than relying on bit-identical arithmetic.
• Seeds are not enough. numpy.random.default_rng(seed) is reproducible across versions for the same bit generator, but numpy.random.seed plus the legacy global state is not guaranteed across major versions. Use explicit Generator objects and record the bit generator name in the fingerprint.
• Reproducibility across architectures. ARM and x86 can differ in floating-point contraction. If releases are produced on one architecture and verified on another, pin the architecture too, or accept a tolerance-based rather than hash-based check.

FAQ # permalink

Does this matter if the release is noised anyway?

Yes, and more so. The suppression decisions — which cells fall below $k$ — are made by geometric predicates, and a different predicate result changes which cells are published. The noise is reproducible from a seed; the geometry is not, unless pinned.

Is a container digest sufficient on its own?

Almost. It pins everything except what the container downloads at runtime, so combine it with a fingerprint assertion at pipeline start so a runtime pip install is caught rather than silently accepted.

Should I pin to exact versions or allow patch updates?

Exact, for anything that produces a published release. Patch releases of GEOS have changed predicate behaviour on degenerate geometry, which is precisely the geometry that boundary-adjacent points present.

How long do I need to keep an environment reproducible?

As long as the release is public plus the period in which someone might challenge it. In practice that means archiving the container image, not just the lockfile — conda channels and package indexes remove old builds.

Related # permalink

• Python Spatial Privacy Toolkit — the stack being pinned
• Automating Spatial Privacy Checks in CI — where the fingerprint assertion runs
• Preserving Topology When Fuzzing Points Near Boundaries — the predicates a GEOS bump moves
• Gating Releases on a Re-identification Risk Budget — the gate that must be reproducible to be meaningful
• Tracking Epsilon Spend in a Release Ledger — recording the fingerprint alongside the charge

← Back to Python Spatial Privacy Toolkit