fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep

The first vision pass OCR'd every large image and, because the "transcribe
the table" prompt makes qwen render ANY image as a markdown table, it turned
60 product UI screenshots into junk "tables" in the corpus. In practice the
only image-only DATA in this corpus is the release-schedule matrix — the
upgrade-compatibility matrices etc. are already native HTML tables, no OCR
needed.

- Add BundleSpec.ocr (default False); the runner passes the OCR-enabling
  session only for bundles that opt in. Set ocr=True only on
  morpheus_release_schedule.
- Purge the 60 screenshot cache entries (keep the release-schedule one). The
  next --force refresh regenerates every other page's .md WITHOUT the bogus
  "## Image transcriptions" section, cleaning the deployed corpus + index.
- Update vision.py / refresh.yml docs to the allowlist model.

Also add scripts/vision_report.py — a self-contained HTML report pairing each
OCR'd source image with its transcription (uncertain-first) for accuracy
inspection. Regenerate anytime from corpus/.vision-cache/.

Verified: release schedule still transcribes (cache hit); cache is back to 1
entry; other bundles now pass no session so their images are left alone.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1

scrape/bundles.py

@@ -62,6 +62,11 @@ class BundleSpec:
     platform: str | None = None
     language: str = "en-US"
     source_url: str | None = None   # overrides the default support.hpe.com URL
+    # Vision-OCR this bundle's images? Default False. Image-only data lives in
+    # only a handful of docs (the release-schedule matrix); every other image
+    # in the corpus is a product UI screenshot that OCR would just add noise
+    # for. So OCR is an explicit per-bundle opt-in, not a corpus-wide sweep.
+    ocr: bool = False
 
 
 # Pinned baseline bundles. docIds confirmed by probing the portal for
@@ -101,7 +106,7 @@ BUNDLES: list[BundleSpec] = [
     # the matrix text comes from the VISION_OCR pass (see scrape/vision.py).
     # version=None — it spans all major streams, not one release.
     BundleSpec("morpheus_release_schedule",    "sf000111242en_us", "HPE Morpheus Software Release Schedule",
-               None, "Release Schedule", "single"),
+               None, "Release Schedule", "single", ocr=True),
 ]
 
 
@@ -180,6 +185,7 @@ def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
             "dates": {},
             "landing_page": spec.doc_id,
             "source_url": spec.source_url or f"https://www.hpe.com/psnow/doc/{spec.doc_id}",
+            "ocr": spec.ocr,
         }
 
     abstract_html = _get(s, f"{API}/{spec.doc_id}", expect_json=False)
@@ -211,6 +217,7 @@ def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
         "dates": {"Published": meta.get("published", "")},
         "landing_page": landing,
         "source_url": spec.source_url or DOC_URL.format(doc_id=spec.doc_id),
+        "ocr": spec.ocr,
     }
 
 

scrape/runner.py

@@ -212,7 +212,10 @@ def scrape_toc_bundle(s: requests.Session, bundle: dict, force: bool, concurrenc
         page_html = fetch_toc_page(s, doc_id, entry.page_id)
         if not page_html:
             return False
-        body_md = html_to_md(page_html, s)
+        # Only pass the session (which enables image OCR) for bundles that
+        # opt in — image-only data lives in a few docs; the rest are UI
+        # screenshots that OCR would just turn into noise.
+        body_md = html_to_md(page_html, s if bundle.get("ocr") else None)
         sidecar = {
             "bundle_id": slug,
             "page_id": entry.page_id,
@@ -243,7 +246,7 @@ def scrape_single_bundle(s: requests.Session, bundle: dict, force: bool) -> int:
     if not html:
         print(f"  ! {slug}: empty body", file=sys.stderr)
         return 0
-    body_md = html_to_md(html, s)
+    body_md = html_to_md(html, s if bundle.get("ocr") else None)
     sidecar = {
         "bundle_id": slug,
         "page_id": doc_id,

scrape/vision.py

@@ -27,13 +27,17 @@ Reliability (the whole point of this module):
     table wrong does so identically every run — so every transcription also
     ships with a "verify against the source image" caveat.
 
-Cost control (a full --force re-scrape re-touches every page weekly):
+Scope: OCR is a per-bundle opt-in (BundleSpec.ocr in scrape/bundles.py), NOT
+a corpus-wide sweep. Image-only data is rare — only the release-schedule
+matrix in this corpus; every other image is a product UI screenshot that OCR
+would just turn into a noise "table". So only allowlisted bundles reach this
+module; the runner passes no session for the rest.
+
+Cost control (secondary, now that scope is an allowlist):
   - Content-hash cache in corpus/.vision-cache/ — each unique image is
-    OCR'd once, ever, and the result is committed so CI reuses it. The same
-    diagram shared across version bundles collapses to one OCR.
-  - VISION_MAX_NEW bounds NEW OCRs per run so the first run can't balloon
-    into hours; the cache fills incrementally over subsequent refreshes.
-    Deferred images are logged (never silently dropped).
+    OCR'd once, ever, and the result is committed so CI reuses it.
+  - VISION_MAX_NEW still bounds NEW OCRs per run as a safety net; with the
+    allowlist it rarely binds. Deferred images are logged, never dropped.
 
 Every failure path degrades to None — a down endpoint, a timeout, a decode
 error — so the scrape never blocks on vision. Defaults target the git.jpaul.io