fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep
The first vision pass OCR'd every large image and, because the "transcribe the table" prompt makes qwen render ANY image as a markdown table, it turned 60 product UI screenshots into junk "tables" in the corpus. In practice the only image-only DATA in this corpus is the release-schedule matrix — the upgrade-compatibility matrices etc. are already native HTML tables, no OCR needed. - Add BundleSpec.ocr (default False); the runner passes the OCR-enabling session only for bundles that opt in. Set ocr=True only on morpheus_release_schedule. - Purge the 60 screenshot cache entries (keep the release-schedule one). The next --force refresh regenerates every other page's .md WITHOUT the bogus "## Image transcriptions" section, cleaning the deployed corpus + index. - Update vision.py / refresh.yml docs to the allowlist model. Also add scripts/vision_report.py — a self-contained HTML report pairing each OCR'd source image with its transcription (uncertain-first) for accuracy inspection. Regenerate anytime from corpus/.vision-cache/. Verified: release schedule still transcribes (cache hit); cache is back to 1 entry; other bundles now pass no session so their images are left alone. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
+10
-6
@@ -27,13 +27,17 @@ Reliability (the whole point of this module):
|
||||
table wrong does so identically every run — so every transcription also
|
||||
ships with a "verify against the source image" caveat.
|
||||
|
||||
Cost control (a full --force re-scrape re-touches every page weekly):
|
||||
Scope: OCR is a per-bundle opt-in (BundleSpec.ocr in scrape/bundles.py), NOT
|
||||
a corpus-wide sweep. Image-only data is rare — only the release-schedule
|
||||
matrix in this corpus; every other image is a product UI screenshot that OCR
|
||||
would just turn into a noise "table". So only allowlisted bundles reach this
|
||||
module; the runner passes no session for the rest.
|
||||
|
||||
Cost control (secondary, now that scope is an allowlist):
|
||||
- Content-hash cache in corpus/.vision-cache/ — each unique image is
|
||||
OCR'd once, ever, and the result is committed so CI reuses it. The same
|
||||
diagram shared across version bundles collapses to one OCR.
|
||||
- VISION_MAX_NEW bounds NEW OCRs per run so the first run can't balloon
|
||||
into hours; the cache fills incrementally over subsequent refreshes.
|
||||
Deferred images are logged (never silently dropped).
|
||||
OCR'd once, ever, and the result is committed so CI reuses it.
|
||||
- VISION_MAX_NEW still bounds NEW OCRs per run as a safety net; with the
|
||||
allowlist it rarely binds. Deferred images are logged, never dropped.
|
||||
|
||||
Every failure path degrades to None — a down endpoint, a timeout, a decode
|
||||
error — so the scrape never blocks on vision. Defaults target the git.jpaul.io
|
||||
|
||||
Reference in New Issue
Block a user