fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep
The first vision pass OCR'd every large image and, because the "transcribe the table" prompt makes qwen render ANY image as a markdown table, it turned 60 product UI screenshots into junk "tables" in the corpus. In practice the only image-only DATA in this corpus is the release-schedule matrix — the upgrade-compatibility matrices etc. are already native HTML tables, no OCR needed. - Add BundleSpec.ocr (default False); the runner passes the OCR-enabling session only for bundles that opt in. Set ocr=True only on morpheus_release_schedule. - Purge the 60 screenshot cache entries (keep the release-schedule one). The next --force refresh regenerates every other page's .md WITHOUT the bogus "## Image transcriptions" section, cleaning the deployed corpus + index. - Update vision.py / refresh.yml docs to the allowlist model. Also add scripts/vision_report.py — a self-contained HTML report pairing each OCR'd source image with its transcription (uncertain-first) for accuracy inspection. Regenerate anytime from corpus/.vision-cache/. Verified: release schedule still transcribes (cache hit); cache is back to 1 entry; other bundles now pass no session so their images are left alone. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
+8
-1
@@ -62,6 +62,11 @@ class BundleSpec:
|
||||
platform: str | None = None
|
||||
language: str = "en-US"
|
||||
source_url: str | None = None # overrides the default support.hpe.com URL
|
||||
# Vision-OCR this bundle's images? Default False. Image-only data lives in
|
||||
# only a handful of docs (the release-schedule matrix); every other image
|
||||
# in the corpus is a product UI screenshot that OCR would just add noise
|
||||
# for. So OCR is an explicit per-bundle opt-in, not a corpus-wide sweep.
|
||||
ocr: bool = False
|
||||
|
||||
|
||||
# Pinned baseline bundles. docIds confirmed by probing the portal for
|
||||
@@ -101,7 +106,7 @@ BUNDLES: list[BundleSpec] = [
|
||||
# the matrix text comes from the VISION_OCR pass (see scrape/vision.py).
|
||||
# version=None — it spans all major streams, not one release.
|
||||
BundleSpec("morpheus_release_schedule", "sf000111242en_us", "HPE Morpheus Software Release Schedule",
|
||||
None, "Release Schedule", "single"),
|
||||
None, "Release Schedule", "single", ocr=True),
|
||||
]
|
||||
|
||||
|
||||
@@ -180,6 +185,7 @@ def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
|
||||
"dates": {},
|
||||
"landing_page": spec.doc_id,
|
||||
"source_url": spec.source_url or f"https://www.hpe.com/psnow/doc/{spec.doc_id}",
|
||||
"ocr": spec.ocr,
|
||||
}
|
||||
|
||||
abstract_html = _get(s, f"{API}/{spec.doc_id}", expect_json=False)
|
||||
@@ -211,6 +217,7 @@ def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
|
||||
"dates": {"Published": meta.get("published", "")},
|
||||
"landing_page": landing,
|
||||
"source_url": spec.source_url or DOC_URL.format(doc_id=spec.doc_id),
|
||||
"ocr": spec.ocr,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user