Validating a Foundation Model Pipeline for Breast Cancer Histopathology: UNI, UMAP, and Spatial QC

A whole-slide image (WSI) doesn't look like much to a neural network. It's a gigapixel file, often 100,000 by 100,000 pixels and tens of gigabytes on disk, and no architecture can ingest that as a single tensor. So the field works around it: slice the slide into thousands of small tiles, encode each tile into a compact feature vector, and treat the whole slide as a bag of those vectors. This post walks through the encoding step (UNI) and the validation checks that come right after it, in the context of a real pipeline predicting breast cancer biomarkers (ER, PR, HER2, Ki-67) from H&E-stained WSIs.

Patch-based preprocessing: why tiling is unavoidable

Tiling is the standard workaround for gigapixel inputs in digital pathology. In this pipeline, each slide is segmented for tissue (via HistoQC), then patched into 256×256 px tiles at 20x magnification using CLAM's preprocessing utilities. The output isn't raw image crops, it's coordinate metadata: an .h5 file per slide holding the (x, y) pixel location of every patch, referencing back into the source .svs file. That coordinate-based design is compact, but it also means the entire spatial correctness of the pipeline hinges on those coordinates being right. If tiling drifts, transposes an axis, or picks up background instead of tissue, nothing downstream will throw an error. It'll just quietly train on garbage.

UNI: a self-supervised encoder for histopathology

UNI is a foundation model for computational pathology from the Mahmood Lab (Brigham and Women's Hospital / Harvard), published in 2024. Functionally, it's a feature extractor: feed it one tile, get back a 1,024-dimensional embedding that encodes the tile's morphology, cellularity, and staining characteristics.

The training regime is what makes it a foundation model rather than a bespoke classifier. UNI was pretrained via DINOv2, a self-supervised learning framework, on upwards of 100 million tiles sampled from more than 100,000 WSIs across roughly 20 tissue types. No tile-level labels were involved in this pretraining stage; the model learned visual representations by solving pretext tasks on the raw pixels, analogous to how large language models acquire syntax and semantics from unlabeled text corpora. The backbone is a ViT-L/16, a large-capacity Vision Transformer.

Using a pretrained encoder instead of training end-to-end on your own cohort is a transfer learning move, and it's close to mandatory here: a few hundred slides is nowhere near enough data to learn robust visual representations from scratch without severe overfitting. UNI supplies a frozen, general-purpose embedding space; a lightweight, task-specific head gets trained on top of it using the small labeled dataset you actually have.

Running inference across every tile in a cohort produces two artifacts per slide: the .h5 file (coordinates plus raw feature array) and a .pt file, a serialized PyTorch tensor of shape [N, 1024]. At scale, this step is the most compute- and storage-intensive stage in the pipeline; a few hundred slides can easily generate tens of gigabytes of embeddings and consume the bulk of your available GPU budget.

Why validation isn't optional

Silent failure is the operative risk here. Coordinate corruption, axis transposition, batch-specific staining or scanner artifacts, or an encoder producing degenerate embeddings, none of these throw exceptions. They just propagate downstream into whatever gets trained on top, and the resulting model can still report a plausible-looking AUC while having learned something spurious. Three checks, run before any classifier training, cover most of the failure surface.

Coordinate scatter plot. Patch (x, y) coordinates from the .h5 file, plotted directly with no image behind them, purely as a geometric sanity check. A tissue-like, organic silhouette confirms coordinates are internally consistent; a rectangular block or random scatter flags a tiling bug.

Coordinate overlay on the thumbnail. The same coordinates, rendered on top of the slide's actual thumbnail via OpenSlide. This is a strictly stronger check: it verifies the patches don't just form a plausible outline in isolation, they're spatially registered to the real tissue in the source image. A shape can pass the coordinate-only check and still fail here if it's offset or mirrored relative to the WSI.

UMAP over the embedding space. UMAP (Uniform Manifold Approximation and Projection) is a nonlinear dimensionality reduction algorithm. It projects each tile's 1,024-dimensional UNI embedding down to two dimensions while approximately preserving local neighborhood structure, so tiles UNI considers similar cluster together and dissimilar ones separate out. A well-behaved projection shows morphologically coherent groupings; a formless, structureless cloud would suggest the encoder is emitting noise rather than signal. Coloring points by extraction batch or session adds a batch-effect diagnostic: if batches interleave rather than forming isolated clusters, there's no evidence of a technical (non-biological) confound in the feature space.

Dependency stack per check, and the rationale

Each check's requirements are dictated by the file format it reads plus whatever computation runs on top of it.

• Coordinate scatter: h5py (HDF5 I/O), numpy, matplotlib.
• Coordinate overlay: the above, plus openslide-python, which wraps the OpenSlide C library, the only practical way to read .svs pyramidal, multi-resolution image files. On Windows, this also requires the compiled OpenSlide binaries linked in separately at runtime, since the Python bindings alone don't ship the underlying library.
• UMAP: torch (tensor deserialization for .pt files), numpy, umap-learn (the actual manifold learning implementation), matplotlib, and typically pandas for grouping and labeling by batch or slide ID during plotting.

Taxonomy: where does this actually sit in the ML landscape?

Worth being precise here, since the answer isn't uniform across the pipeline.

UNI's pretraining is self-supervised learning, a subcategory of unsupervised learning: no human-annotated labels are involved, and the model derives its representation space entirely from the structure of the raw pixel data.

UMAP is also unsupervised, and in this context it's purely diagnostic. It has no notion of biomarker status; it never contributes to a prediction. It exists solely to let a human visually audit whether the feature space is structured or degenerate.

The downstream classifier, CLAM (Clustering-constrained Attention Multiple Instance Learning), is weakly supervised. It receives exactly one label per slide (ER-positive, say) but has no ground truth for any of the thousands of individual tiles composing that slide. It's a multiple instance learning (MIL) formulation: the model treats each slide as a "bag" of tile-level instances, learns an attention distribution over that bag, and aggregates into a single slide-level prediction. MIL is the dominant paradigm in computational pathology precisely because exhaustive tile-level annotation is clinically and economically infeasible at scale.

Ki-67 doesn't go through any of this. As a proliferation index, it's quantified via direct cell counting on IHC-stained slides using QuPath with the StarDist cell-detection model, not a slide-level MIL classifier.

Clinical relevance in breast cancer

ER, PR, and HER2 status are the biomarkers that determine treatment stratification (endocrine therapy, HER2-targeted agents such as trastuzumab, chemotherapy regimens), and they're conventionally assessed via immunohistochemistry (IHC), a separate stain from the H&E slides most public digital pathology cohorts contain. The clinical value proposition of a pipeline like this is predicting IHC-equivalent status directly from routine H&E, which is cheaper and far more widely available. This is also why the project maintains a strict separation between two cohorts: TCGA, which is H&E-only, so any ER/PR/HER2 output on it is a model prediction, not a confirmed pathology result, and the project's own CMDN slides, which carry real pathologist-scored IHC labels usable as ground truth. Conflating predicted and confirmed labels anywhere in a results table or writeup would constitute a genuine methodological error, not a cosmetic one.