# scrape/

Product-specific. **You implement this for each product.** The
template gives you the contract; the extraction logic depends on
the upstream doc portal.

See `PLAN.md` Phase 1 for the corpus layout the rest of the pipeline
expects.

## What you write

At minimum, two scripts:

### `scrape/bundles.py`

Discovers the upstream portal's bundle catalog and writes
`bundles.json` at the repo root. One entry per bundle (versioned doc
set) with the schema in PLAN.md.

```bash
python -m scrape.bundles
```

### `scrape/runner.py`

Scrapes the pages of each bundle (or a single bundle with `--bundle
<slug>`). Writes:

- `corpus/<bundle_id>/<page_id>.md` — extracted markdown body
- `corpus/<bundle_id>/<page_id>.json` — per-page metadata sidecar

```bash
python -m scrape.runner --all --force --concurrency 6
python -m scrape.runner --bundle Admin.VC.HTML.10.9
```

## Tips

- **Sniff before you scrape.** Almost every modern doc portal is an
  SPA that calls a backend API. Open the browser's Network tab,
  click around, find the underlying JSON. Scraping the API is 10×
  cheaper and 100× more reliable than scraping the rendered HTML.
- **Idempotent re-scrapes.** Without `--force`, the runner should
  skip pages already on disk so a resume doesn't have to re-fetch
  everything. With `--force`, re-fetch every page — that's the
  weekly cron mode that catches edits.
- **Respect the portal.** Backoff on 429s. Set a recognizable
  user-agent so the portal owner can identify you if they want to.
- **Whitespace normalize.** Markdown that round-trips through HTML
  often has extra blank lines. Normalize to a single blank between
  paragraphs so diffs are clean (the changelog summary and digest
  tools care about line counts).

## What's already reusable

`scrape/changelog.py` is fully product-agnostic and ready to use
as-is. It walks `git diff --name-status` output to produce a
structured summary, and walks `git log` for the digest history
(Phase 13).