fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep

The first vision pass OCR'd every large image and, because the "transcribe
the table" prompt makes qwen render ANY image as a markdown table, it turned
60 product UI screenshots into junk "tables" in the corpus. In practice the
only image-only DATA in this corpus is the release-schedule matrix — the
upgrade-compatibility matrices etc. are already native HTML tables, no OCR
needed.

- Add BundleSpec.ocr (default False); the runner passes the OCR-enabling
  session only for bundles that opt in. Set ocr=True only on
  morpheus_release_schedule.
- Purge the 60 screenshot cache entries (keep the release-schedule one). The
  next --force refresh regenerates every other page's .md WITHOUT the bogus
  "## Image transcriptions" section, cleaning the deployed corpus + index.
- Update vision.py / refresh.yml docs to the allowlist model.

Also add scripts/vision_report.py — a self-contained HTML report pairing each
OCR'd source image with its transcription (uncertain-first) for accuracy
inspection. Regenerate anytime from corpus/.vision-cache/.

Verified: release schedule still transcribes (cache hit); cache is back to 1
entry; other bundles now pass no session so their images are left alone.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
2026-07-23 13:31:09 -04:00
co-authored by Claude Opus 4.8
parent 6686b033e6
commit b31121d90c
66 changed files with 221 additions and 446 deletions
+20 -10
View File
@@ -14,7 +14,8 @@
"Published": "February 2026"
},
"landing_page": "GUID-709AAADB-A9C1-40B6-AD22-958EE7E6F312",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007510en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007510en_us",
"ocr": false
},
{
"slug": "morpheus_user_manual_8_1_1",
@@ -31,7 +32,8 @@
"Published": "March 2026"
},
"landing_page": "GUID-709AAADB-A9C1-40B6-AD22-958EE7E6F312",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007621en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007621en_us",
"ocr": false
},
{
"slug": "morpheus_user_manual_8_1_2",
@@ -48,7 +50,8 @@
"Published": "April 2026"
},
"landing_page": "GUID-709AAADB-A9C1-40B6-AD22-958EE7E6F312",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007732en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007732en_us",
"ocr": false
},
{
"slug": "morpheus_release_notes_8_1_0",
@@ -65,7 +68,8 @@
"Published": "February 2026"
},
"landing_page": "sd00007496en_us",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007496en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007496en_us",
"ocr": false
},
{
"slug": "morpheus_release_notes_8_1_1",
@@ -82,7 +86,8 @@
"Published": "March 2026"
},
"landing_page": "sd00007610en_us",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007610en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007610en_us",
"ocr": false
},
{
"slug": "morpheus_release_notes_8_1_2",
@@ -99,7 +104,8 @@
"Published": "April 2026"
},
"landing_page": "sd00007733en_us",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007733en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007733en_us",
"ocr": false
},
{
"slug": "morpheus_user_manual_9_0_0",
@@ -116,7 +122,8 @@
"Published": "May 2026"
},
"landing_page": "GUID-709AAADB-A9C1-40B6-AD22-958EE7E6F312",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00008014en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00008014en_us",
"ocr": false
},
{
"slug": "morpheus_release_notes_9_0_0",
@@ -133,7 +140,8 @@
"Published": "May 2026"
},
"landing_page": "sd00008017en_us",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00008017en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sd00008017en_us",
"ocr": false
},
{
"slug": "morpheus_quickspecs",
@@ -148,7 +156,8 @@
"abstract": "",
"dates": {},
"landing_page": "a50009231enw",
"source_url": "https://www.hpe.com/psnow/doc/a50009231enw"
"source_url": "https://www.hpe.com/psnow/doc/a50009231enw",
"ocr": false
},
{
"slug": "morpheus_release_schedule",
@@ -165,6 +174,7 @@
"Published": ""
},
"landing_page": "sf000111242en_us",
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sf000111242en_us"
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sf000111242en_us",
"ocr": true
}
]