hvm-docs/scrape

scrape/

Product-specific. You implement this for each product. The template gives you the contract; the extraction logic depends on the upstream doc portal.

See PLAN.md Phase 1 for the corpus layout the rest of the pipeline expects.

What you write

At minimum, two scripts:

scrape/bundles.py

Discovers the upstream portal's bundle catalog and writes bundles.json at the repo root. One entry per bundle (versioned doc set) with the schema in PLAN.md.

python -m scrape.bundles

scrape/runner.py

Scrapes the pages of each bundle (or a single bundle with --bundle <slug>). Writes:

• corpus/<bundle_id>/<page_id>.md — extracted markdown body
• corpus/<bundle_id>/<page_id>.json — per-page metadata sidecar

python -m scrape.runner --all --force --concurrency 6
python -m scrape.runner --bundle Admin.VC.HTML.10.9

Tips

• Sniff before you scrape. Almost every modern doc portal is an SPA that calls a backend API. Open the browser's Network tab, click around, find the underlying JSON. Scraping the API is 10× cheaper and 100× more reliable than scraping the rendered HTML.
• Idempotent re-scrapes. Without --force, the runner should skip pages already on disk so a resume doesn't have to re-fetch everything. With --force, re-fetch every page — that's the weekly cron mode that catches edits.
• Respect the portal. Backoff on 429s. Set a recognizable user-agent so the portal owner can identify you if they want to.
• Whitespace normalize. Markdown that round-trips through HTML often has extra blank lines. Normalize to a single blank between paragraphs so diffs are clean (the changelog summary and digest tools care about line counts).

What's already reusable

scrape/changelog.py is fully product-agnostic and ready to use as-is. It walks git diff --name-status output to produce a structured summary, and walks git log for the digest history (Phase 13).