SurgGround — Build Log — Duc-Anh Nguyen

What it is

At 1 fps a 2-hour procedure is ~7,200 frames — impossible for any vision-language model to see in one pass. The project's spine is fitting a 30–150 minute procedure into a small VLM (InternVL3-2B) at low token cost via a trained TemporalConnector with always-on token reduction, then localizing precisely inside it with a hierarchical-coarse-to-fine or retrieval regime depending on length. Three research bets sit on top of that engineering: abstention-shaped offline RL that penalizes confident wrong answers as much as missed ones, a procedure-graph constraint that makes an impossible phase order unrepresentable, and a clean compression-vs-hierarchy-vs-retrieval comparison across the length range — all trained and evaluated on two owned RTX 4090s, no cluster required.

InternVL3-2B
QLoRA
Offline RL (RAFT/DPO)
Temporal grounding
Procedure graphs
Calibrated abstention
GraSP
MultiBypass140

Status now

2 / 12

phases merged (P0 scaffold, P2 task+metric harness) + P1 data code merged

passing tests, up from 40 at scaffold

training datasets landing on the GPU boxes right now

2 × 4090

owned RTX 4090s, running training + eval in parallel

Phases

Phase	What	Status
P0	Scaffold, config, pure modules	merged
P1	Six dataset parsers, decode, splits	code merged · data landing
P2	Task construction + metric modules	merged
P3	Zero-shot baseline harness — first real numbers	next
P4	TemporalConnector + QLoRA SFT — the complete-result gate	planned
P5–P9	Long-video regimes, procedure-graph decode, offline RL, reliability, OOD	planned
P10–P11	Optional JEPA pretraining, demo + report	planned

Datasets, right now

The two genuinely long (~2-hour-class) procedures in the plan — GraSP and MultiBypass140 — are the headline datasets; Cholec80 is the fast-iteration control.
Dataset	State
Cholec80	landed
GraSP	landed
MultiBypass140	downloading (4 of 6 zips)

Real work, not just a plan

The first real download of MultiBypass140 on the training box filled a 242 GB disk to zero bytes free and produced not one usable frame.

What happened

The download script located each surgical centre's videos at a fixed path inside its zip. Real S3 archives from this dataset don't agree on layout — some wrap their contents in an extra directory, some don't — so every "wrapped" zip's ~40–90 GB of video matched no path the script knew to check. It was never extracted to frames, and — since the script only deleted a video after successfully extracting it — never deleted either. It just accumulated, silently, until a disk-full error hit an unattended background job mid-write and the job hung indefinitely.

Fix

Rewrote the extraction step to search for every directory literally named videos after each zip, wherever it landed, and infer the surgical centre from the matched path itself instead of assuming where it would be. Verified against a synthetic reproduction of both zip layouts, then against the real zip that had triggered the incident — found, extracted, and cleaned up correctly. No real data was lost: the frame output directory on the training GPU was never even touched, since the buggy step never got that far.

What's next

Finish landing all three datasets on the training box, build the frame index, and spot-check parser output against the raw annotations — the actual P1 exit criteria.
P3 — zero-shot baselines. Run ≥2 open video-LLMs on grounding, phase segmentation and QA across both the short and long-video regimes. Expected finding: zero-shot grounding on a 2.5-hour procedure is near-random — quantifying that gap is the point.
P4 — the first complete result. QLoRA-train the TemporalConnector on the 2B backbone; target ≥10-point grounding improvement over the best zero-shot baseline and a non-trivial number on the 2.5-hour dataset. This alone stands as a result even if nothing after it ships in time.

Full technical spec, every design decision, and the day-to-day status log are in the repo.