fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep
The first vision pass OCR'd every large image and, because the "transcribe the table" prompt makes qwen render ANY image as a markdown table, it turned 60 product UI screenshots into junk "tables" in the corpus. In practice the only image-only DATA in this corpus is the release-schedule matrix — the upgrade-compatibility matrices etc. are already native HTML tables, no OCR needed. - Add BundleSpec.ocr (default False); the runner passes the OCR-enabling session only for bundles that opt in. Set ocr=True only on morpheus_release_schedule. - Purge the 60 screenshot cache entries (keep the release-schedule one). The next --force refresh regenerates every other page's .md WITHOUT the bogus "## Image transcriptions" section, cleaning the deployed corpus + index. - Update vision.py / refresh.yml docs to the allowlist model. Also add scripts/vision_report.py — a self-contained HTML report pairing each OCR'd source image with its transcription (uncertain-first) for accuracy inspection. Regenerate anytime from corpus/.vision-cache/. Verified: release schedule still transcribes (cache hit); cache is back to 1 entry; other bundles now pass no session so their images are left alone. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
+5
-2
@@ -212,7 +212,10 @@ def scrape_toc_bundle(s: requests.Session, bundle: dict, force: bool, concurrenc
|
||||
page_html = fetch_toc_page(s, doc_id, entry.page_id)
|
||||
if not page_html:
|
||||
return False
|
||||
body_md = html_to_md(page_html, s)
|
||||
# Only pass the session (which enables image OCR) for bundles that
|
||||
# opt in — image-only data lives in a few docs; the rest are UI
|
||||
# screenshots that OCR would just turn into noise.
|
||||
body_md = html_to_md(page_html, s if bundle.get("ocr") else None)
|
||||
sidecar = {
|
||||
"bundle_id": slug,
|
||||
"page_id": entry.page_id,
|
||||
@@ -243,7 +246,7 @@ def scrape_single_bundle(s: requests.Session, bundle: dict, force: bool) -> int:
|
||||
if not html:
|
||||
print(f" ! {slug}: empty body", file=sys.stderr)
|
||||
return 0
|
||||
body_md = html_to_md(html, s)
|
||||
body_md = html_to_md(html, s if bundle.get("ocr") else None)
|
||||
sidecar = {
|
||||
"bundle_id": slug,
|
||||
"page_id": doc_id,
|
||||
|
||||
Reference in New Issue
Block a user