Does in-domain SSL improve calibration?

The question

For four model settings — {LeJEPA, DINO v1} × {SigLIP-2 ViT-B/16 init, ImageNet ViT-B/16 init} — measure zero-shot calibration on HyperKvasir (weighted k-NN on frozen features, k = 20), then repeat after in-domain self-supervised pretraining on GastroNet-5M, and report the change in ECE and NLL across five downstream tasks (four classification, one zero-shot polyp segmentation).

• PyTorch Lightning
• LeJEPA (SIGReg)
• DINO v1
• SigLIP-2 / ImageNet ViT-B/16
• k-NN calibration
• Temperature scaling
• SLURM / H100 & A100
• Weights & Biases
• WebDataset (LRZ-direct ingest)

Training specs

Every setting below ran to completion — full hyperparameters, loss/LR curves and per-step timing for all four are logged live to the W&B project. The table is the fixed part shared by every run; what varies is only the objective and the initialization.

Zero-shot baseline — before any adaptation

Each backbone frozen at its public initialization, no in-domain SSL yet. Weighted k-NN over a 60% reference split, temperature fit on a 15% cal split, metrics on the held-out query split (~2,650 images per classification task; 900 query masks for segmentation). The point of this baseline is the contrast it draws with the post-SSL tables below.

ImageNet init beats SigLIP-2 on every task — the opposite of the pre-registered guess that a web-scale contrastive model would transfer better. For endoscopy k-NN it does not: supervised ImageNet features are more linearly separable here, and their patch tokens carry far more local structure (segmentation Dice 0.67 vs 0.31).

The tract task is flagged — confirmed structural, not leakage. A dedicated ontology audit found 23 of 23 HyperKvasir findings are tract-exclusive (each occurs in upper-GI or lower-GI images only, never both), so upper-vs-lower-GI is recoverable from coarse image content almost by construction, independent of the accuracy number itself. Recorded as an interpretation caveat rather than treated as a leak.

Temperature scaling is not uniformly helpful: for ImageNet, T > 1 and post-T ECE sometimes rises (findings: .048 → .111) — the k-NN “logits” are not true logits, and a scalar fit on a 15% cal split can over-correct. The pre-vs-post-SSL delta below is the intended read, not the absolute baseline value.

The headline finding

Result table — Δ = post-SSL − pre-SSL

Weighted k-NN, k = 20, on a perceptual-hash-deduplicated split (near-duplicate endoscopy frames between the k-NN reference set and the held-out query set were removed first — 17% of query images). Negative ΔECE / ΔNLL = SSL improved that metric. Green = SSL helped, red = SSL hurt, grey = within noise.

DINO v1 / ImageNet init — effective rank 51.7, no collapse

DINO v1 / SigLIP-2 init — effective rank 46.7, no collapse

LeJEPA / SigLIP-2 init — effective rank 25.8, no collapse

LeJEPA / ImageNet init — severe regression, effective rank 33.9 (not collapsed)

The story across all four settings: DINO is the reliable adapter — it improved calibration and/or segmentation from both initializations, with no accuracy cost anywhere. LeJEPA is powerful but init-sensitive — from SigLIP-2 (room to improve) it trades accuracy for markedly better calibration and a strong segmentation gain; from ImageNet (already excellent) it is destructive, since the aggressive regularizer has nothing to gain and a great deal to lose. Findings and segmentation are where SSL matters most for the three well-behaved settings; the near-saturated tract/pathology tasks barely move either way. Practical takeaway: if the deployment init is a strong supervised backbone, prefer a gentle self-distillation objective (DINO) for in-domain adaptation, and reserve an aggressive distributional regularizer (LeJEPA) for weaker starting points.

Compute

Case study — the LR = 0 bug

The first full-scale training job started on an A100 the morning of 2026-09-08. Watching its scalar dashboard live caught something the smoke tests could not: it was not training at all.

Two independent bugs compounded:

1. The framework's own step counter never advanced. The training step called the raw optimizer's .step() instead of the wrapped one — a workaround for an intermittent CPU hang on a login node, left in place on GPU where it was never needed. On GPU that means the step counter stays at 0: the cosine LR schedule, keyed on that counter, sits at its warmup-step-zero value — LR ≡ 0, so every optimizer step is a no-op — and the checkpoint trigger, keyed on the same counter, never fires either. A crash-and-resume would have restarted from scratch, forever.
2. The data pipeline's notion of “one epoch” was 12× too long. An epoch-length setting meant for the whole pipeline had been applied per data-loading worker instead, so with 12 parallel workers one epoch became roughly 5 hours instead of the intended length. The other checkpoint trigger, keyed on epoch-end, would not have fired inside the job's time limit either.

Together: the run could produce nothing usable, and would have looped indefinitely, silently, across every restart.

The GPU smoke test had passed because a handful of steps at a tiny batch size sits entirely inside LR warmup — the rate is near-zero there whether or not the counter moves, and the smoke test only checks that a checkpoint file appears and that resuming from it works, which the epoch-end trigger did handle at that scale. The bug only showed up under a full-scale run, which is exactly why the run was being watched live rather than fired and checked the next day.