justin
a0727da8da
Two new bundles:
* hvm_qualification_matrix (sd00006551en_us) — the "Qualification Matrix
for HVM Clusters Managed by HPE Morpheus Software". Single TOC bundle,
2 pages (parent + content). The content page is ~100 KB of HTML
containing five tables: Server Hardware Support, Storage Hardware
Support, Independent Software Vendor (ISV) Support, Hypervisor OS
Compatibility and Interoperability Matrix, and Guest OS. Scraped via
the same /hpesc/public/api/document/{docId}/render endpoint as every
other bundle on support.hpe.com — the API returns server-rendered
DITA HTML, so no JS/SPA shenanigans.
* hvm_quickspecs (a50004260enw) — HPE Morpheus VM Essentials Software
QuickSpecs, Version 4 (02-Feb-2026). SKUs: S5Q81AAE (1-yr per Socket
E-LTU), S5Q82AAE (3-yr), S5Q83AAE (5-yr); each includes Tech Care
Essentials. QuickSpecs lives at www.hpe.com (not support.hpe.com),
which drops connections at the edge for non-browser TLS fingerprints —
verified 2026-05-22 against curl, wget, urllib, and Anthropic's
WebFetch (all = 0 bytes / connection timeout in headers). Bypassed
here via curl_cffi impersonating Chrome 120's JA3/JA4 fingerprint.
HTTP 200, 255 KB on first try, all four sections + all three SKUs
cleanly parseable from the server-rendered HTML.
New module scrape/quickspecs.py drives the live fetch + parse for any
hvm_*_quickspecs bundle. CSS selectors taken from the captured DOM:
.lr-right-rail hpe-highlights-container .collateral-content
— one block per H3 section
h3.txto-title — section title
div.txto-description — section body
uc-table.uc-table-polaris — SKU and version-history tables
On any live failure the parser falls back to a committed HTML fixture
at scrape/quickspecs/<doc_id>.html so the build never breaks on a
transient edge hiccup.
scrape/runner.py learned a new mode "html-file" that dispatches to
scrape.quickspecs; bundles.py extended with an optional source_url on
BundleSpec for cases where the page lives outside support.hpe.com.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scrape/
Product-specific. You implement this for each product. The template gives you the contract; the extraction logic depends on the upstream doc portal.
See PLAN.md Phase 1 for the corpus layout the rest of the pipeline
expects.
What you write
At minimum, two scripts:
scrape/bundles.py
Discovers the upstream portal's bundle catalog and writes
bundles.json at the repo root. One entry per bundle (versioned doc
set) with the schema in PLAN.md.
python -m scrape.bundles
scrape/runner.py
Scrapes the pages of each bundle (or a single bundle with --bundle <slug>). Writes:
corpus/<bundle_id>/<page_id>.md— extracted markdown bodycorpus/<bundle_id>/<page_id>.json— per-page metadata sidecar
python -m scrape.runner --all --force --concurrency 6
python -m scrape.runner --bundle Admin.VC.HTML.10.9
Tips
- Sniff before you scrape. Almost every modern doc portal is an SPA that calls a backend API. Open the browser's Network tab, click around, find the underlying JSON. Scraping the API is 10× cheaper and 100× more reliable than scraping the rendered HTML.
- Idempotent re-scrapes. Without
--force, the runner should skip pages already on disk so a resume doesn't have to re-fetch everything. With--force, re-fetch every page — that's the weekly cron mode that catches edits. - Respect the portal. Backoff on 429s. Set a recognizable user-agent so the portal owner can identify you if they want to.
- Whitespace normalize. Markdown that round-trips through HTML often has extra blank lines. Normalize to a single blank between paragraphs so diffs are clean (the changelog summary and digest tools care about line counts).
What's already reusable
scrape/changelog.py is fully product-agnostic and ready to use
as-is. It walks git diff --name-status output to produce a
structured summary, and walks git log for the digest history
(Phase 13).