fix(scrape): make vision OCR a per-bundle allowlist, not a corpus sweep (#6)

Co-authored-by: claude <[email protected]>
This commit was merged in pull request #6.
This commit is contained in:
2026-08-06 11:39:47 -04:00
committed by claude
parent 43d891968e
commit 08c4b4b71d
66 changed files with 223 additions and 447 deletions
+10 -6
View File
@@ -27,13 +27,17 @@ Reliability (the whole point of this module):
table wrong does so identically every run — so every transcription also
ships with a "verify against the source image" caveat.
Cost control (a full --force re-scrape re-touches every page weekly):
Scope: OCR is a per-bundle opt-in (BundleSpec.ocr in scrape/bundles.py), NOT
a corpus-wide sweep. Image-only data is rare — only the release-schedule
matrix in this corpus; every other image is a product UI screenshot that OCR
would just turn into a noise "table". So only allowlisted bundles reach this
module; the runner passes no session for the rest.
Cost control (secondary, now that scope is an allowlist):
- Content-hash cache in corpus/.vision-cache/ — each unique image is
OCR'd once, ever, and the result is committed so CI reuses it. The same
diagram shared across version bundles collapses to one OCR.
- VISION_MAX_NEW bounds NEW OCRs per run so the first run can't balloon
into hours; the cache fills incrementally over subsequent refreshes.
Deferred images are logged (never silently dropped).
OCR'd once, ever, and the result is committed so CI reuses it.
- VISION_MAX_NEW still bounds NEW OCRs per run as a safety net; with the
allowlist it rarely binds. Deferred images are logged, never dropped.
Every failure path degrades to None — a down endpoint, a timeout, a decode
error — so the scrape never blocks on vision. Defaults target the git.jpaul.io