fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep
The first vision pass OCR'd every large image and, because the "transcribe the table" prompt makes qwen render ANY image as a markdown table, it turned 60 product UI screenshots into junk "tables" in the corpus. In practice the only image-only DATA in this corpus is the release-schedule matrix — the upgrade-compatibility matrices etc. are already native HTML tables, no OCR needed. - Add BundleSpec.ocr (default False); the runner passes the OCR-enabling session only for bundles that opt in. Set ocr=True only on morpheus_release_schedule. - Purge the 60 screenshot cache entries (keep the release-schedule one). The next --force refresh regenerates every other page's .md WITHOUT the bogus "## Image transcriptions" section, cleaning the deployed corpus + index. - Update vision.py / refresh.yml docs to the allowlist model. Also add scripts/vision_report.py — a self-contained HTML report pairing each OCR'd source image with its transcription (uncertain-first) for accuracy inspection. Regenerate anytime from corpus/.vision-cache/. Verified: release schedule still transcribes (cache hit); cache is back to 1 entry; other bundles now pass no session so their images are left alone. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
@@ -37,13 +37,13 @@ env:
|
||||
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436,http://192.168.0.125:11434,http://192.168.0.126:11434
|
||||
EMBED_MODEL: nomic-embed-text
|
||||
|
||||
# Vision OCR of image-only pages (support/lifecycle matrices etc. — see
|
||||
# scrape/vision.py). Uses the host's primary Ollama on :11434, the only one
|
||||
# with a vision model; the embed pool above is nomic-only. qwen2.5vl:7b
|
||||
# scored best on the release-schedule matrix (55-56/56, ~18s, GPU-resident,
|
||||
# prompt-robust). Content-hash cached in corpus/.vision-cache/; VISION_MAX_NEW
|
||||
# bounds NEW OCRs per run so the cache fills incrementally instead of a
|
||||
# multi-hour first pass. Any failure degrades to the pre-vision behavior.
|
||||
# Vision OCR of image-only data pages (the release-schedule matrix — see
|
||||
# scrape/vision.py). OCR is a per-bundle opt-in (BundleSpec.ocr), so this
|
||||
# only runs on allowlisted docs, NOT every screenshot in the corpus. Uses
|
||||
# the host's primary Ollama on :11434, the only one with a vision model
|
||||
# (the embed pool above is nomic-only). qwen2.5vl:7b scored best on the
|
||||
# matrix (55-56/56, ~18s, GPU-resident). Any failure degrades to the
|
||||
# pre-vision behavior.
|
||||
VISION_OCR: "1"
|
||||
VISION_URL: http://192.168.0.2:11434
|
||||
VISION_MODEL: qwen2.5vl:7b
|
||||
|
||||
Reference in New Issue
Block a user