feat(scrape): vision OCR of image-only doc pages (support matrices)
Some HPE pages carry information only inside an image. The Morpheus release schedule / support-lifecycle matrix (sf000111242en_us) is the case that surfaced this: the entire "which version ships when, when a stream hits Maintenance/EOL" table is a JPEG, so html_to_md dropped every value and retrieval could never answer "when does 8.x.x reach end of life". New scrape/vision.py (opt-in via VISION_OCR) transcribes qualifying images with a local Ollama vision model and appends a labeled markdown block to the page so the text is chunked, embedded and retrieved. Model/prompt chosen against ground truth on the matrix (14x4 cells): - qwen2.5vl:7b + a bare "transcribe the table" prompt: 55-56/56, ~18s, GPU-resident, and prompt-ROBUST (accurate without hand-tuning). - gemma3:12b needs an exact "read row by row, keep cells aligned" prompt or it shifts a column by one row; qwen2.5vl:32b is no more accurate and 5x slower (CPU spill). Both documented in the code. - Note: self-consistency (2 samples must agree) only catches RANDOM flakiness — a wrong read is stable across seeds — so every block also ships a "verify against the source image" caveat. Reliability/cost: - content-hash cache in corpus/.vision-cache/ (committed) — each unique image OCR'd once ever, shared across version bundles. - VISION_MAX_NEW bounds NEW OCRs per run so the first pass can't balloon into hours; the cache fills incrementally. Deferred count is logged. - every failure path degrades to the pre-vision behavior; never blocks the scrape. Also: - add the morpheus_release_schedule bundle (sf single-doc) + 2 eval golden queries for it. - fetch_single_doc: title falls back to the bundle title (not docId) when a page has no <h1> (sf solution articles). - Pillow in requirements-vision.txt (scrape-only; kept out of the server image), installed + VISION_* wired into refresh.yml. Verified locally: matrix transcribes to the exact 14-row table; second run is a cache hit (ocr=0). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
This commit is contained in:
Claude Opus 4.8
+7
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"sha": "4cb0e044a29be128f264ec1c1d65f704e2ce3488d8d1c8ddf1d89e8cc97c9cfc",
|
||||
"src": "https://support.hpe.com/hpesc/public/api/document/sf000111242en_us/ka2Kh0000008hBd_Procedure__c_0EMKh000006RMP6.jpg?v=3.0",
|
||||
"model": "qwen2.5vl:7b",
|
||||
"certain": true,
|
||||
"markdown": "| Release Window | 7.x.x | 8.x.x | 9.x.x | 10.x.x |\n|----------------|-------|-------|-------|--------|\n| January 2026 | Maintenance | 8.0.13 | | |\n| February 2026 | | 8.1.0 | | |\n| March 2026 | | 8.1.1 | | |\n| April 2026 | | 8.1.2 | | |\n| May 2026 | EOL | Maintenance | 9.0.0 | |\n| June 2026 | | | 9.0.1 | |\n| July 2026 | | | 9.0.2 | |\n| August 2026 | | | 9.1.0 | |\n| September 2026 | | | 9.1.1 | |\n| Oct '26 - Apr '27 | | | 9.1.2 - 9.3.2 | |\n| May 2027 | | EOL | Maintenance | 10.0.0 |\n| June 2027 | | | | 10.0.1|\n| July 2027 | | | | 10.0.2|\n| August 2027 | | | | 10.1.0|"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"bundle_id": "morpheus_release_schedule",
|
||||
"page_id": "sf000111242en_us",
|
||||
"title": "HPE Morpheus Software Release Schedule",
|
||||
"ordinal": 1,
|
||||
"parent_title": null,
|
||||
"doc_id": "sf000111242en_us",
|
||||
"version": null,
|
||||
"product": "Release Schedule",
|
||||
"source_url": "https://support.hpe.com/hpesc/public/docDisplay?docId=sf000111242en_us"
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
# Steps
|
||||
|
||||
|
||||
|
||||
HPE Morpheus Software supports each major version for a period of two years. This support window is split between a twelve-month period of active development followed by another twelve-month period of maintenance support. During active maintenance, an upgrade containing new features is released once per quarter (version x.1.x, x.2.x, etc). Each month between the quarterly feature upgrades, we will deliver a maintenance upgrade containing improvements and bug fixes (version x.x.1 and x.x.2). After this twelve-month period of active development ends, a new major upgrade is delivered (version 9.x.x, 10.x.x, etc) and a twelve-month period of maintenance support begins. During maintenance support, there are no scheduled monthly upgrades. However, support services are still available and critical security upgrades will be delivered should the need arise. After this twelve-month period of maintenance support ends, all versions within that major release umbrella will reach end of life. **After the security maintenance window ends and end-of-life is reached, access to Morpheus support services for that major version stream will also end. To receive support, you will be required to upgrade to a supported version.**
|
||||
|
||||
**NOTE:** The table above is meant to be an illustration of the expected release cadence described. It is not necessarily a commitment to release specific version numbers at specific times.
|
||||
|
||||
## Image transcriptions
|
||||
|
||||
> Transcribed from the image above by local vision OCR (qwen2.5vl:7b); verify against the source image.
|
||||
|
||||
| Release Window | 7.x.x | 8.x.x | 9.x.x | 10.x.x |
|
||||
|----------------|-------|-------|-------|--------|
|
||||
| January 2026 | Maintenance | 8.0.13 | | |
|
||||
| February 2026 | | 8.1.0 | | |
|
||||
| March 2026 | | 8.1.1 | | |
|
||||
| April 2026 | | 8.1.2 | | |
|
||||
| May 2026 | EOL | Maintenance | 9.0.0 | |
|
||||
| June 2026 | | | 9.0.1 | |
|
||||
| July 2026 | | | 9.0.2 | |
|
||||
| August 2026 | | | 9.1.0 | |
|
||||
| September 2026 | | | 9.1.1 | |
|
||||
| Oct '26 - Apr '27 | | | 9.1.2 - 9.3.2 | |
|
||||
| May 2027 | | EOL | Maintenance | 10.0.0 |
|
||||
| June 2027 | | | | 10.0.1|
|
||||
| July 2027 | | | | 10.0.2|
|
||||
| August 2027 | | | | 10.1.0|
|
||||
Reference in New Issue
Block a user