fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep #6

2026-07-23T13:31:25-04:00

claude commented

2026-07-23 13:31:25 -04:00

Problem

The first vision pass OCR'd every large image. Because the "transcribe the table" prompt makes qwen render any image as a markdown table, it turned 60 product UI screenshots into junk "tables" in the corpus. In reality the only image-only data here is the release-schedule matrix — the upgrade-compatibility matrices are already native HTML tables (no OCR needed).

Fix — allowlist instead of sweep

BundleSpec.ocr (default False): the runner passes the OCR-enabling session only for opted-in bundles. Set ocr=True only on morpheus_release_schedule.
Purged the 60 screenshot cache entries (kept the release-schedule one). The next --force refresh regenerates every other page's .md without the bogus ## Image transcriptions section — cleaning the deployed corpus + index.
Docs (vision.py / refresh.yml) updated to the allowlist model.
Adds scripts/vision_report.py — the self-contained image-vs-transcription inspection report.

Verified

Release schedule still transcribes (cache hit); cache back to 1 entry; other bundles pass no session so their images are left alone. After merge, the refresh will strip the 60 screenshot transcriptions corpus-wide and redeploy.

## Problem The first vision pass OCR'd every large image. Because the "transcribe the table" prompt makes qwen render **any** image as a markdown table, it turned **60 product UI screenshots into junk "tables"** in the corpus. In reality the only image-only *data* here is the **release-schedule matrix** — the upgrade-compatibility matrices are already native HTML tables (no OCR needed). ## Fix — allowlist instead of sweep - **`BundleSpec.ocr`** (default `False`): the runner passes the OCR-enabling session only for opted-in bundles. Set `ocr=True` **only** on `morpheus_release_schedule`. - **Purged the 60 screenshot cache entries** (kept the release-schedule one). The next `--force` refresh regenerates every other page's `.md` **without** the bogus `## Image transcriptions` section — cleaning the deployed corpus + index. - Docs (vision.py / refresh.yml) updated to the allowlist model. - Adds **`scripts/vision_report.py`** — the self-contained image-vs-transcription inspection report. ## Verified Release schedule still transcribes (cache hit); cache back to 1 entry; other bundles pass no session so their images are left alone. After merge, the refresh will strip the 60 screenshot transcriptions corpus-wide and redeploy.

claude added 1 commit 2026-08-06 11:39:36 -04:00

fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep 1286cb7fdd

The first vision pass OCR'd every large image and, because the "transcribe
the table" prompt makes qwen render ANY image as a markdown table, it turned
60 product UI screenshots into junk "tables" in the corpus. In practice the
only image-only DATA in this corpus is the release-schedule matrix — the
upgrade-compatibility matrices etc. are already native HTML tables, no OCR
needed.

- Add BundleSpec.ocr (default False); the runner passes the OCR-enabling
  session only for bundles that opt in. Set ocr=True only on
  morpheus_release_schedule.
- Purge the 60 screenshot cache entries (keep the release-schedule one). The
  next --force refresh regenerates every other page's .md WITHOUT the bogus
  "## Image transcriptions" section, cleaning the deployed corpus + index.
- Update vision.py / refresh.yml docs to the allowlist model.

Also add scripts/vision_report.py — a self-contained HTML report pairing each
OCR'd source image with its transcription (uncertain-first) for accuracy
inspection. Regenerate anytime from corpus/.vision-cache/.

Verified: release schedule still transcribes (cache hit); cache is back to 1
entry; other bundles now pass no session so their images are left alone.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1

claude force-pushed fix/vision-ocr-allowlist from b31121d90c to 1286cb7fdd

2026-08-06 11:39:36 -04:00

Compare

claude merged commit 08c4b4b71d into main

2026-08-06 11:39:47 -04:00

claude deleted branch fix/vision-ocr-allowlist

2026-08-06 11:39:47 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: justin/morpheus-docs#6