scrape: add Qualification Matrix + QuickSpecs bundles (live curl_cffi for HPE www)
Two new bundles:
* hvm_qualification_matrix (sd00006551en_us) — the "Qualification Matrix
for HVM Clusters Managed by HPE Morpheus Software". Single TOC bundle,
2 pages (parent + content). The content page is ~100 KB of HTML
containing five tables: Server Hardware Support, Storage Hardware
Support, Independent Software Vendor (ISV) Support, Hypervisor OS
Compatibility and Interoperability Matrix, and Guest OS. Scraped via
the same /hpesc/public/api/document/{docId}/render endpoint as every
other bundle on support.hpe.com — the API returns server-rendered
DITA HTML, so no JS/SPA shenanigans.
* hvm_quickspecs (a50004260enw) — HPE Morpheus VM Essentials Software
QuickSpecs, Version 4 (02-Feb-2026). SKUs: S5Q81AAE (1-yr per Socket
E-LTU), S5Q82AAE (3-yr), S5Q83AAE (5-yr); each includes Tech Care
Essentials. QuickSpecs lives at www.hpe.com (not support.hpe.com),
which drops connections at the edge for non-browser TLS fingerprints —
verified 2026-05-22 against curl, wget, urllib, and Anthropic's
WebFetch (all = 0 bytes / connection timeout in headers). Bypassed
here via curl_cffi impersonating Chrome 120's JA3/JA4 fingerprint.
HTTP 200, 255 KB on first try, all four sections + all three SKUs
cleanly parseable from the server-rendered HTML.
New module scrape/quickspecs.py drives the live fetch + parse for any
hvm_*_quickspecs bundle. CSS selectors taken from the captured DOM:
.lr-right-rail hpe-highlights-container .collateral-content
— one block per H3 section
h3.txto-title — section title
div.txto-description — section body
uc-table.uc-table-polaris — SKU and version-history tables
On any live failure the parser falls back to a committed HTML fixture
at scrape/quickspecs/<doc_id>.html so the build never breaks on a
transient edge hiccup.
scrape/runner.py learned a new mode "html-file" that dispatches to
scrape.quickspecs; bundles.py extended with an optional source_url on
BundleSpec for cases where the page lives outside support.hpe.com.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+29
-2
@@ -39,9 +39,10 @@ class BundleSpec:
|
||||
title: str
|
||||
version: str | None
|
||||
product: str # e.g. "User Manual", "Release Notes", "Deployment Guide"
|
||||
mode: str # "toc" or "single"
|
||||
mode: str # "toc", "single", or "html-file" (committed fixture under scrape/quickspecs/)
|
||||
platform: str | None = None
|
||||
language: str = "en-US"
|
||||
source_url: str | None = None # overrides the default support.hpe.com URL
|
||||
|
||||
|
||||
# Declared bundles. Versions confirmed 2026-05-22 by probing the docId
|
||||
@@ -54,6 +55,14 @@ BUNDLES: list[BundleSpec] = [
|
||||
BundleSpec("hvm_release_notes_8_1_1", "sd00007609en_us", "HPE Morpheus VM Essentials Software Release Notes", "8.1.1", "Release Notes", "single"),
|
||||
BundleSpec("hvm_release_notes_8_1_2", "sd00007734en_us", "HPE Morpheus VM Essentials Software Release Notes", "8.1.2", "Release Notes", "single"),
|
||||
BundleSpec("hvm_deployment_guide", "sd00007332en_us", "HPE Morpheus VM Essentials Deployment Guide", None, "Deployment Guide","toc"),
|
||||
BundleSpec("hvm_qualification_matrix","sd00006551en_us", "Qualification Matrix for HVM Clusters Managed by HPE Morpheus Software", None, "Qualification Matrix", "toc"),
|
||||
# QuickSpecs is a static-HTML fixture (www.hpe.com edge drops automated
|
||||
# connections — see scrape/quickspecs/README.md). doc_id = the QuickSpecs
|
||||
# PSNow ref (a50004260enw). page_count is 1; source_url points at the
|
||||
# public PSNow URL.
|
||||
BundleSpec("hvm_quickspecs", "a50004260enw", "HPE Morpheus VM Essentials Software QuickSpecs",
|
||||
"v4-2026-02-02", "QuickSpecs", "html-file",
|
||||
source_url="https://www.hpe.com/psnow/doc/a50004260enw"),
|
||||
]
|
||||
|
||||
|
||||
@@ -118,6 +127,24 @@ def _parse_abstract(html: str) -> dict[str, str]:
|
||||
|
||||
|
||||
def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
|
||||
# html-file bundles are static fixtures — no upstream fetch.
|
||||
if spec.mode == "html-file":
|
||||
return {
|
||||
"slug": spec.slug,
|
||||
"doc_id": spec.doc_id,
|
||||
"title": spec.title,
|
||||
"version": spec.version,
|
||||
"platform": spec.platform,
|
||||
"product": spec.product,
|
||||
"language": spec.language,
|
||||
"page_count": 1,
|
||||
"mode": "html-file",
|
||||
"abstract": "",
|
||||
"dates": {},
|
||||
"landing_page": spec.doc_id,
|
||||
"source_url": spec.source_url or f"https://www.hpe.com/psnow/doc/{spec.doc_id}",
|
||||
}
|
||||
|
||||
abstract_html = _get(s, f"{API}/{spec.doc_id}", expect_json=False)
|
||||
meta = _parse_abstract(abstract_html or "")
|
||||
|
||||
@@ -146,7 +173,7 @@ def discover_bundle(s: requests.Session, spec: BundleSpec) -> dict[str, Any]:
|
||||
"abstract": meta.get("abstract", ""),
|
||||
"dates": {"Published": meta.get("published", "")},
|
||||
"landing_page": landing,
|
||||
"source_url": DOC_URL.format(doc_id=spec.doc_id),
|
||||
"source_url": spec.source_url or DOC_URL.format(doc_id=spec.doc_id),
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user