← Duc-Anh Nguyen · Curated-subset results
For {LeJEPA, DINO v1} × ImageNet ViT-B/16 init, measure zero-shot calibration on HyperKvasir, then repeat after self-supervised pretraining on the complete ~4.9M-image raw GastroNet-5M corpus — all 506 portal archives, streamed directly rather than pre-curated. DINO/ImageNet comes out with zero regressions on accuracy, balanced-accuracy, AUROC or Dice on any task, and is promoted as the resulting foundation-model checkpoint.
Before training, the win condition was fixed in advance (to rule out picking whichever result looks best after the fact): SSL pretraining must not make the model worse, on average, at telling classes apart or segmenting polyps — specifically, no regression on mean balanced-accuracy across the four HyperKvasir classification tasks, or on segmentation Dice, versus the pre-SSL baseline. DINO/ImageNet passes; LeJEPA/ImageNet fails it. DINO's checkpoint is the resulting foundation model, logged as a versioned W&B Artifact for further fine-tuning on other downstream tasks.
All five come from HyperKvasir and share the same frozen-feature, weighted-k-NN protocol (k=20 for classification, k=10 for segmentation) — nothing here is a trained classifier head, so every number below is a direct read of how separable the frozen embedding space already is.
| hkv_findings 20-way | The finest-grained label HyperKvasir has: the
specific endoscopic finding in the frame (e.g. a polyp type, an esophagitis grade, a named anatomical
landmark). Actually 23 classes in the raw dataset; 3 are dropped for having fewer than 12 labeled images
(hemorrhoids, ileum, ulcerative-colitis-grade-1-2), leaving 20.
The hardest task — most classes, finest distinctions — and where SSL moves the needle most.
2,196 query images. |
| hkv_category 4-way | A coarser bucket each finding belongs to: anatomical-landmarks, pathological-findings, quality-of-mucosal-views, or therapeutic-interventions. 2,201 query images. |
| hkv_pathology 2-way | Collapses hkv_category further,
to the clinically loaded question: is this frame something a clinician should flag
(pathological-findings), or not (everything else — landmarks, prep-quality shots,
therapeutic-intervention frames)? Not the same grouping as "any of the 20 findings" — e.g.
cecum, pylorus, z-line (landmarks) and
bbps-0-1/bbps-2-3 (bowel-prep quality) are not pathological.
2,201 query images. |
| hkv_tract 2-way | Upper GI (esophagus/stomach/duodenum, from gastroscopy) vs. lower GI (colon/rectum, from colonoscopy) — a coarse but non-trivial which-procedure question, and already near-saturated pre-SSL (99.4% balanced-accuracy). 2,201 query images. |
| hkv_seg pixel-level | Zero-shot polyp segmentation: dense ViT patch-token features, k-NN label transfer from a 100-image support set, scored by Dice/mIoU against ground-truth masks on 900 held-out frames. The only task that tests where in the frame something is, not just what the frame shows. |
Two datasets with different levels of certainty. HyperKvasir (the evaluation data) was audited exhaustively: every one of its 10,619 images was decoded, hashed and counted. GastroNet-5M (the self-supervised training data, 4,823,974 images) is far too large to decode twice, so its integrity and redundancy were measured on a random sample of 24 of the 493 zip archives (226,256 images, 4.7% of the corpus; seed 0) and extrapolated with uncertainty ranges; the archive and image counts themselves are exact, read from the full manifest. Redundancy is measured with a 64-bit perceptual hash (pHash): two frames are “near-duplicates” if their hashes differ in at most 6 of 64 bits — the same threshold the evaluation split uses, so the two datasets are directly comparable. A pHash match means visually near-identical frames; there are no patient or procedure IDs, so it cannot prove they come from the same procedure.
| HyperKvasir step | images | share of 10,619 | what happened |
|---|---|---|---|
Rows in image-labels.csv | 10,662 | – | the published label table |
| Image files present locally | 10,619 | 100.0% | 43 labelled rows have no image file (28 esophagitis-b-d, 15 esophagitis-a): the local copy is incomplete; they cannot be used |
| Decodable (strict decoder) | 10,619 | 100.0% | 0 corrupt, 0 truncated, all RGB; nothing to repair |
| Exact byte duplicates | 0 | 0.0% | none found |
| Near-duplicate frames (pHash ≤ 6 bits) | 1,819 | 17.1% | 2,079 images (19.6%) have at least one near-duplicate; kept in the dataset, but see next row |
| Removed from calibration/query as near-duplicates of a reference image | 727 | 6.8% | calibration −274 of 1,592 (17.2%), query −453 of 2,654 (17.1%): leakage control |
| Scored in the 4 classification tasks | 9,892 | 93.2% | reference 6,373 + calibration 1,318 + query 2,201 |
| … of which in the 20-class findings task | 9,867 | 92.9% | 3 classes with < 12 images (26 images in total) are excluded |
| Polyp segmentation set (separate files) | 1,000 | – | 100 support + 900 query pairs; all decodable, 0 empty masks, 0 image/mask size mismatches |
| GastroNet-5M | count | share | how it was established |
|---|---|---|---|
| Portal archives | 506 | 100% | all used; the manifest build fails if any archive is missing or unreadable (exact) |
| Images (frames) per epoch | 4,823,974 | 100% | exact manifest count; per-archive min 2,256, median 9,914, max 13,287; 3 epochs = 14,471,922 image presentations |
| Frames removed before training | 0 | 0% | none: the raw stream is used as stored (no resize, re-encode or de-duplication) |
| Sample inspected | 226,256 | 4.7% | 24 random zip archives; the 13 .7z archives (2.6%) were not sampled |
| check | HyperKvasir (all 10,619) | GastroNet-5M (sample of 226,256) | treatment |
|---|---|---|---|
| Unreadable / corrupt | 0 (0.00%) | 0 (0.0000%) 95% upper bound 0.0013% ≈ 64 of 4,823,974 frames | If a frame cannot be decoded, or its archive member is missing, a deterministic blank 224×224 sample takes its place, so step counts and checkpoint resume stay exact. Never triggered: 0 failures in about 13.4M decode attempts of the winning run, and 0 in the sample. |
| Truncated PNG/JPEG | 0 | 0 | Decoded with truncation tolerance. Caveat: this means the “0 failures” above does not prove there are none. Before this fix, both training runs crashed on “image file is truncated” right after each resume, which indicates a truncated PNG exists somewhere in the full corpus (probably one image hit repeatedly; inferred from the logs, not verified). None was in the sample. |
| Not RGB / grayscale-like | 0 non-RGB | 0 non-RGB; 26 near-grayscale (0.011%) | Converted to RGB; nothing removed. |
| Exact byte duplicates | 0 | 75 (0.033%) | HyperKvasir: none to remove. GastroNet: kept. |
| Identical pHash | 335 (3.2%) | 205 (0.09%) 95% CI 0.05–0.14% | HyperKvasir evaluation: a calibration/query image within 6 bits of any reference image is dropped (leakage control, see above); near-duplicates inside the scoring splits remain (41 of 3,519, 1.2%). GastroNet: deliberately kept. The stream is unlabelled, so near-duplicates cannot leak a label, but they do mean the effective number of distinct frames is lower than 4.82M by roughly the percentage shown, and the most redundant archives are visited as often as the rest. |
| Near-duplicate, ≤ 2 bits | 950 (8.9%) | 640 (0.28%) 95% CI 0.12–0.52% | |
| Near-duplicate, ≤ 6 bits | 1,819 (17.1%) | 13,500 (5.97%) 95% CI 3.22–10.62% → about 288,000 frames of the corpus (range 155,000–512,000) | |
| Polyp-mask set (1,000 pairs) | near-dup. 30 (3.0%) | – | Only 1 of 900 query images is within 2 bits of a support image, so support-to-query leakage is negligible; no filtering applied. |
Greyed classes are excluded from the benchmark (fewer than 12 images). The largest class has 1,148 images, the smallest kept class 28: an imbalance ratio of 41:1.
Red: classes with fewer than 20 query images (6 of 20; the smallest has 6). Per-class accuracy on these is very noisy, so the benchmark reports balanced accuracy over all classes with the caveat that a handful of images can move a class by tens of points.
The filter is concentrated in three classes: bowel-prep scores (bbps-2-3, bbps-0-1) and impacted-stool. Most of their held-out images were near-copies of reference images and were removed (probably because they come from runs of very similar consecutive frames; the dataset has no procedure IDs to confirm this). Near-copies of reference frames inflate k-NN accuracy, so keeping them would likely have made these classes look easier than they are.
| task | classes | images | largest class | imbalance max:min | effective classes | chance bal-acc | always-majority accuracy (query) | pre-SSL bal-acc |
|---|---|---|---|---|---|---|---|---|
| findings | 20 | 10,593 | 10.8% | 41.0:1 | 14.3 | 5% | 11.3% | 60.6% |
| category | 4 | 10,619 | 38.6% | 2.1:1 | 3.8 | 25% | 45.1% | 94.0% |
| pathology | 2 | 10,619 | 75.5% | 3.1:1 | 1.7 | 50% | 71.7% | 90.1% |
| tract | 2 | 10,619 | 67.9% | 2.1:1 | 1.9 | 50% | 62.2% | 99.4% |
Median polyp covers 11.4% of the frame (middle half 5.9–21.0%, 5th–95th percentile 2.5–40.8%). 200 of 1,000 masks are below 5% of the frame, where a few misplaced patches move Dice a lot: a hard regime for 14×14 patch-token transfer.
363 of 506 archives hold 9.5–10k images; a small number are much smaller (27 archives hold under 8,000 images, together 3.5% of all frames). All 506 are used in every epoch.
Frames are not square and not a fixed size (85% are 512 pixels high, widths vary), stored as RGB PNG (mean 382 KB per frame). The training transform resizes and crops them to the network input; no re-encoding is applied to the stored data.
Central 90% of frames lie between grey level 73 and 127. Practically no unusable frames: 4 near-black and 1 near-white frame in 226,256; 28 frames are more than half dark pixels (borders, dark views). So blank-filling as a fallback has almost nothing to compensate for.
Red = within 6 bits, i.e. near-duplicate: 7.2% of sampled frames have such a neighbour (HyperKvasir, for comparison, 19.6%). Most frames sit 10–14 bits from their nearest neighbour; a typical frame is genuinely different from every other in its archive.
Typical archives have 3.4% (median); archive 0483 has 53%. This heterogeneity is why the corpus-level estimate is a range (3.22–10.62%, bootstrap over archives), not a point.
.7z archives were not sampled, and a whole-archive defect could be missed. The zero-corruption bound assumes frames are independent.Result table — Δ = post-SSL − pre-SSL effective rank 14.3 at completion, stable across all 3 epochs (14.0 → 14.1 → 14.3), no collapse
| task | ΔECE ew | ΔECE·T | ΔNLL | Δacc | Δbal-acc | ΔAUROC | ΔDice |
|---|---|---|---|---|---|---|---|
| findings (20-way) | −.0255 | −.0250 | −.052 | +.016 | +.029 | +.010 | – |
| category (4-way) | −.0117 | −.0024 | −.009 | +.004 | +.001 | +.002 | – |
| pathology (2-way) | −.0039 | +.0113 | +.007 | +.002 | +.002 | +.001 | – |
| tract (2-way) | .0000 | −.0038 | −.010 | +.002 | +.001 | 0 | – |
| segmentation | +.0012 | – | −.039 | – | – | – | +.079 |
Accuracy, balanced-accuracy and AUROC are all non-negative across every one of the four classification tasks, and the largest single gain in the study is segmentation Dice +0.079 (0.667 → 0.745). The one exception is a single ECE cell (temperature-scaled, pathology task, +.0113) — a slightly worse calibration reading on an already near-saturated task, not a discrimination metric, and not part of the selection rule below. Effective rank stayed stable throughout (14.0 → 14.1 → 14.3, collapse reads ~1), and the run was clean end to end — zero decode failures across 13.4M images attempted from the raw, uncurated corpus.
Full metrics — absolute values pre-SSL → post-SSL, then Δ in percentage points and Δ as a percent change relative to the pre-SSL value
| task / metric | pre-SSL → post-SSL | Δ (pp) | Δ (%) |
|---|---|---|---|
| findings (20-way, n=2,196) | |||
| accuracy | 82.42% → 84.02% | +1.59 | +1.9% |
| balanced-accuracy | 60.65% → 63.57% | +2.93 | +4.8% |
| AUROC | 94.60% → 95.56% | +0.96 | +1.0% |
| ECE (equal-width) | 5.30% → 2.75% | −2.55 | −48.1% |
| ECE (temp-scaled) | 12.25% → 9.75% | −2.50 | −20.4% |
| NLL | 0.6294 → 0.5776 | −0.0517 | −8.2% |
| category (4-way, n=2,201) | |||
| accuracy | 93.09% → 93.46% | +0.36 | +0.4% |
| balanced-accuracy | 93.97% → 94.10% | +0.13 | +0.1% |
| AUROC | 98.87% → 99.05% | +0.18 | +0.2% |
| ECE (equal-width) | 2.94% → 1.77% | −1.17 | −39.6% |
| ECE (temp-scaled) | 4.65% → 4.41% | −0.24 | −5.1% |
| NLL | 0.2037 → 0.1948 | −0.0089 | −4.4% |
| pathology (2-way, n=2,201) | |||
| accuracy | 93.37% → 93.59% | +0.23 | +0.2% |
| balanced-accuracy | 90.12% → 90.32% | +0.21 | +0.2% |
| AUROC | 97.98% → 98.04% | +0.06 | +0.1% |
| ECE (equal-width) | 1.97% → 1.58% | −0.39 | −19.9% |
| ECE (temp-scaled) | 4.15% → 5.28% | +1.13 | +27.2% |
| NLL | 0.1747 → 0.1813 | +0.0066 | +3.8% |
| tract (2-way, n=2,201) | |||
| accuracy | 99.36% → 99.55% | +0.18 | +0.2% |
| balanced-accuracy | 99.44% → 99.56% | +0.12 | +0.1% |
| AUROC | 99.89% → 99.93% | +0.04 | +0.0% |
| ECE (equal-width) | 0.23% → 0.23% | +0.00 | +1.0% |
| ECE (temp-scaled) | 0.77% → 0.39% | −0.38 | −49.8% |
| NLL | 0.0277 → 0.0177 | −0.0100 | −36.2% |
| segmentation (n=900 query, 100 support) | |||
| Dice | 66.67% → 74.53% | +7.86 | +11.8% |
| mIoU | 54.06% → 62.61% | +8.56 | +15.8% |
| ECE (equal-width) | 0.93% → 1.05% | +0.12 | +13.4% |
| NLL | 0.2384 → 0.1998 | −0.0385 | −16.2% |
Result table — Δ = post-SSL − pre-SSL effective rank 13.2 at completion, stable across all 3 epochs (12.1 → 13.0 → 13.2), no collapse
| task | ΔECE ew | ΔECE·T | ΔNLL | Δacc | Δbal-acc | ΔAUROC | ΔDice |
|---|---|---|---|---|---|---|---|
| findings (20-way) | −.0208 | −.0378 | +.169 | −.052 | −.037 | −.026 | – |
| category (4-way) | −.0050 | −.0141 | +.091 | −.034 | −.035 | −.008 | – |
| pathology (2-way) | −.0022 | −.0139 | +.084 | −.030 | −.052 | −.018 | – |
| tract (2-way) | +.0033 | −.0017 | +.011 | −.004 | −.006 | −.001 | – |
| segmentation | +.0048 | – | +.063 | – | – | – | −.079 |
Calibration improves (ECE down on 4 of 5 tasks) while accuracy consistently drops: every one of the 4 classification tasks regresses on both accuracy and balanced-accuracy, and segmentation Dice is down 0.079. Effective rank stayed healthy throughout (13.2, collapse reads ~1), so this isn't representation collapse — it reads as LeJEPA's SIGReg term reshaping an already class-structured supervised embedding faster than it can be put to use. The practical takeaway: match the SSL objective's aggressiveness to how much the starting representation actually needs to move — LeJEPA is the wrong tool for an already-strong supervised init; DINO is not.
Full metrics — absolute values pre-SSL → post-SSL, then Δ in percentage points and Δ as a percent change relative to the pre-SSL value
| task / metric | pre-SSL → post-SSL | Δ (pp) | Δ (%) |
|---|---|---|---|
| findings (20-way, n=2,196) | |||
| accuracy | 82.42% → 77.23% | −5.19 | −6.3% |
| balanced-accuracy | 60.65% → 56.90% | −3.74 | −6.2% |
| AUROC | 94.60% → 91.97% | −2.63 | −2.8% |
| ECE (equal-width) | 5.30% → 3.22% | −2.08 | −39.3% |
| ECE (temp-scaled) | 12.25% → 8.47% | −3.78 | −30.8% |
| NLL | 0.6294 → 0.7987 | +0.1693 | +26.9% |
| category (4-way, n=2,201) | |||
| accuracy | 93.09% → 89.69% | −3.41 | −3.7% |
| balanced-accuracy | 93.97% → 90.51% | −3.46 | −3.7% |
| AUROC | 98.87% → 98.10% | −0.77 | −0.8% |
| ECE (equal-width) | 2.94% → 2.44% | −0.50 | −17.0% |
| ECE (temp-scaled) | 4.65% → 3.24% | −1.41 | −30.3% |
| NLL | 0.2037 → 0.2944 | +0.0908 | +44.6% |
| pathology (2-way, n=2,201) | |||
| accuracy | 93.37% → 90.37% | −3.00 | −3.2% |
| balanced-accuracy | 90.12% → 84.91% | −5.21 | −5.8% |
| AUROC | 97.98% → 96.16% | −1.82 | −1.9% |
| ECE (equal-width) | 1.97% → 1.75% | −0.22 | −10.9% |
| ECE (temp-scaled) | 4.15% → 2.75% | −1.39 | −33.6% |
| NLL | 0.1747 → 0.2585 | +0.0838 | +48.0% |
| tract (2-way, n=2,201) | |||
| accuracy | 99.36% → 99.00% | −0.36 | −0.4% |
| balanced-accuracy | 99.44% → 98.82% | −0.62 | −0.6% |
| AUROC | 99.89% → 99.79% | −0.10 | −0.1% |
| ECE (equal-width) | 0.23% → 0.55% | +0.33 | +144.8% |
| ECE (temp-scaled) | 0.77% → 0.60% | −0.17 | −21.7% |
| NLL | 0.0277 → 0.0383 | +0.0106 | +38.2% |
| segmentation (n=900 query, 100 support) | |||
| Dice | 66.67% → 58.77% | −7.91 | −11.9% |
| mIoU | 54.06% → 46.90% | −7.16 | −13.2% |
| ECE (equal-width) | 0.93% → 1.41% | +0.48 | +51.5% |
| NLL | 0.2384 → 0.3015 | +0.0632 | +26.5% |
Rather than a curated tier that resizes/re-encodes/deduplicates every image before training,
this reads the raw portal archives directly, so a small fraction of images are genuinely
corrupt or truncated in the source data (handled by falling back to a blank sample rather than
dropping it, to keep exact step counts — see the codebase for details). Full
hyperparameters, loss/LR curves and per-step timing logged live to
W&B,
tagged s506_raw_e3.
| Backbone | ViT-B/16, 768-d · feature = LayerNorm(mean patch token), non-affine |
| Initialization | ImageNet-1k supervised (see the 581k-image pilot for the SigLIP-2 comparison that ruled it out here) |
| SSL objectives | LeJEPA (SIGReg isotropy + variance hinge + stop-grad prediction) · DINO v1 (centering + sharpening) |
| Corpus | All 506 GastroNet-5M portal archives, ~4.9M images, streamed raw (no local pre-staging) |
| Optimizer | AdamW, cosine schedule, base LR 1e-4 · layer-wise LR decay 0.75 · grad clip 3.0 |
| Batch size · precision | 128 · bf16 mixed, no gradient checkpointing (unnecessary at this batch size on 80–94GB GPUs) |
| Schedule | 3 full passes over the corpus · ~37,688 steps/epoch × 3 = 113,064 total optimizer steps per setting, verified by checkpoint step count |
| Zero-shot eval | Weighted k-NN, k = 20, τ = 0.07, cosine similarity on L2-normalized features |
| Calibration | ECE (15 equal-width + 15 adaptive bins), NLL, Brier · scalar temperature via LBFGS |
| Hardware | Single GPU per run, shared H100 94GB / A100‑SXM4‑80GB cluster, SLURM-scheduled, self-resubmitting across the 48h wall-time limit |
Training data
| Coverage | All 506/506 portal archives (100%) — the manifest build hard-fails if any archive is missing or unreadable, so no archive is silently skipped |
| Images per epoch | 4,823,974, counted directly off the indexed manifest (min 2,256 · max 13,287 · mean ~9,534 images per archive) |
| Corrupt / truncated images | Not dropped — replaced with a deterministic blank sample so step counts and checkpoint resume stay exact. On the winning DINO run: 0 decode failures logged across ~13.4M decode attempts |
| Duplicates | No deduplication is applied to this training corpus — near-duplicate frames from the same procedure are expected and left in, unlike the HyperKvasir eval side, which does run a perceptual-hash dedup to keep near-duplicates from leaking across splits |
| Distribution / class balance | Unlabeled by design, so no class or anatomical-site breakdown exists for this corpus — a documented characteristic of GastroNet-5M is a skew toward upper-GI (gastroscopy) procedures, not something measured in this run |
Would blank-filling a corrupt image instead of dropping it hurt training? In practice here, no measurable effect — at ~0 observed failures out of 13.4M decode attempts, essentially no batch ever actually contained a blank sample, so the design choice never got stress-tested by real data. In principle, if the rate were higher, the failure mode would differ by objective: DINO's multi-crop self-distillation would just see two identical black "views" — a trivial, information-free pair, wasted compute rather than actively harmful, though at high-enough volume the model could start treating solid black as its own valid cluster. LeJEPA is more exposed: its SIGReg term computes batch-level isotropy/variance statistics, and a degenerate blank-image embedding skews those batch statistics directly rather than just averaging out — at a high-enough contamination rate that could measurably distort the isotropy target.
Training telemetry pulled directly from the two runs' W&B history, not re-typed from a summary
DINO and LeJEPA report loss on different scales (DINO's is a cross-entropy-style centering/sharpening loss; LeJEPA's is a SIGReg isotropy + variance + prediction composite), so per the never-dual-axis rule they get separate small-multiple charts below rather than one chart with two y-axes. Both share the same optimizer-step x-axis (0 → 113,064) and the same base LR (1e-4) on an identical cosine schedule, so the LR curve is not duplicated as a third chart.
y = DINO loss (centering/sharpening cross-entropy) · x = optimizer step. 15-step rolling mean, downsampled to ~120 points from the full per-step W&B history. Run d337dbe3837db7e8.
y = LeJEPA loss (SIGReg isotropy + variance hinge + stop-grad prediction, summed) · x = optimizer step. Same 15-step rolling mean and ~120-point downsampling as the DINO chart. Run c2371ea2b6034bac. The composite drops fast in the first few hundred steps then flattens — consistent with SIGReg reshaping the ImageNet-init embedding early and settling.
Effective rank by epoch boundary same axis for both settings — this is the collapse-check evidence behind the "no collapse" note above
Effective rank measures how many dimensions of the 768-d embedding are actually being used, not just declared. It's computed from the singular values σi of a batch of embeddings: normalize them into a probability-like distribution pi = σi / ∑σi, then take the exponential of their entropy, erank = exp(−∑i pi log pi). If every dimension carried equal weight, erank would equal 768; if the embedding collapsed onto a single direction (representation collapse — a real failure mode for self-supervised objectives without a good defense against the trivial solution), erank would drop toward 1. A value in the low teens, stable or climbing, means neither run collapsed.
y = effective rank (embedding dimension count) · x = epoch boundary (global step 37,687 / 75,375 / 113,063). Exact values, no smoothing — only 3 logged points per run. DINO stays higher throughout and keeps climbing; LeJEPA starts lower and closes most of the gap by epoch 3, but neither shows the rank collapsing toward 1.
All three charts pulled fresh from Weights & Biases run history (wandb.Api(), project vlf-zeiss) for runs d337dbe3837db7e8 (DINO/ImageNet) and c2371ea2b6034bac (LeJEPA/ImageNet).