claudeandClaude Opus 4.8
cd30d0dc7a
HVM (HPE Morpheus VM Essentials) 9.0.0 shipped June 2026; the MCP had
nothing about it. Add it and stop hand-adding a docId per release.
scrape/bundles.py now AUTO-DISCOVERS new versions (discover_specs, on by
default): it scans the DocPortal docId range above the highest known
bundle and classifies each abstract. On-by-default in CI's refresh.yml,
so future releases flow into the corpus with no code change. Gotchas
encoded:
- classify on the abstract BODY, not the <h1> (a Release Notes h1 is
just "v9.0.0 Release Notes" with no product name)
- EXCLUDE the interleaved siblings that share the same version string:
Morpheus Enterprise, Morpheus Central, Storage Integration Pack
- failsoft: a scan error logs a warning and falls back to known bundles
- self-healing + idempotent: prior bundles.json persists discoveries;
the scan anchor rolls forward; re-runs add no duplicates
Discovered + scraped for 9.0.0:
- User Manual sd00008058en_us (414 pages; Deployment Guide now folded in)
- Release Notes sd00008079en_us
corpus: +412 pages (9.0), sidecar topic_cluster peers re-linked to 9.0
across the 8.1.x bundles (shared GUIDs). Local reindex: 3645 chunks
(956 are 9.0.0); 9.0 content verified retrievable end-to-end.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MNQRS6F7N9Uwh1SMn3xz2g
scrape/
Product-specific. You implement this for each product. The template gives you the contract; the extraction logic depends on the upstream doc portal.
See PLAN.md Phase 1 for the corpus layout the rest of the pipeline
expects.
What you write
At minimum, two scripts:
scrape/bundles.py
Discovers the upstream portal's bundle catalog and writes
bundles.json at the repo root. One entry per bundle (versioned doc
set) with the schema in PLAN.md.
python -m scrape.bundles
scrape/runner.py
Scrapes the pages of each bundle (or a single bundle with --bundle <slug>). Writes:
corpus/<bundle_id>/<page_id>.md— extracted markdown bodycorpus/<bundle_id>/<page_id>.json— per-page metadata sidecar
python -m scrape.runner --all --force --concurrency 6
python -m scrape.runner --bundle Admin.VC.HTML.10.9
Tips
- Sniff before you scrape. Almost every modern doc portal is an SPA that calls a backend API. Open the browser's Network tab, click around, find the underlying JSON. Scraping the API is 10× cheaper and 100× more reliable than scraping the rendered HTML.
- Idempotent re-scrapes. Without
--force, the runner should skip pages already on disk so a resume doesn't have to re-fetch everything. With--force, re-fetch every page — that's the weekly cron mode that catches edits. - Respect the portal. Backoff on 429s. Set a recognizable user-agent so the portal owner can identify you if they want to.
- Whitespace normalize. Markdown that round-trips through HTML often has extra blank lines. Normalize to a single blank between paragraphs so diffs are clean (the changelog summary and digest tools care about line counts).
What's already reusable
scrape/changelog.py is fully product-agnostic and ready to use
as-is. It walks git diff --name-status output to produce a
structured summary, and walks git log for the digest history
(Phase 13).