fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep

The first vision pass OCR'd every large image and, because the "transcribe
the table" prompt makes qwen render ANY image as a markdown table, it turned
60 product UI screenshots into junk "tables" in the corpus. In practice the
only image-only DATA in this corpus is the release-schedule matrix — the
upgrade-compatibility matrices etc. are already native HTML tables, no OCR
needed.

- Add BundleSpec.ocr (default False); the runner passes the OCR-enabling
  session only for bundles that opt in. Set ocr=True only on
  morpheus_release_schedule.
- Purge the 60 screenshot cache entries (keep the release-schedule one). The
  next --force refresh regenerates every other page's .md WITHOUT the bogus
  "## Image transcriptions" section, cleaning the deployed corpus + index.
- Update vision.py / refresh.yml docs to the allowlist model.

Also add scripts/vision_report.py — a self-contained HTML report pairing each
OCR'd source image with its transcription (uncertain-first) for accuracy
inspection. Regenerate anytime from corpus/.vision-cache/.

Verified: release schedule still transcribes (cache hit); cache is back to 1
entry; other bundles now pass no session so their images are left alone.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
2026-08-06 11:38:50 -04:00
committed by justin
co-authored by Claude Opus 4.8
parent 43d891968e
commit 1286cb7fdd
66 changed files with 223 additions and 447 deletions
+7 -7
View File
@@ -39,13 +39,13 @@ env:
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436,http://192.168.0.125:11434,http://192.168.0.126:11434
EMBED_MODEL: nomic-embed-text
# Vision OCR of image-only pages (support/lifecycle matrices etc. — see
# scrape/vision.py). Uses the host's primary Ollama on :11434, the only one
# with a vision model; the embed pool above is nomic-only. qwen2.5vl:7b
# scored best on the release-schedule matrix (55-56/56, ~18s, GPU-resident,
# prompt-robust). Content-hash cached in corpus/.vision-cache/; VISION_MAX_NEW
# bounds NEW OCRs per run so the cache fills incrementally instead of a
# multi-hour first pass. Any failure degrades to the pre-vision behavior.
# Vision OCR of image-only data pages (the release-schedule matrix — see
# scrape/vision.py). OCR is a per-bundle opt-in (BundleSpec.ocr), so this
# only runs on allowlisted docs, NOT every screenshot in the corpus. Uses
# the host's primary Ollama on :11434, the only one with a vision model
# (the embed pool above is nomic-only). qwen2.5vl:7b scored best on the
# matrix (55-56/56, ~18s, GPU-resident). Any failure degrades to the
# pre-vision behavior.
VISION_OCR: "1"
VISION_URL: http://192.168.0.2:11434
VISION_MODEL: qwen2.5vl:7b