claude
9600235466
Four independent regional brands across IA/IN/IL (variety-identity sources,
each parsed into structured characteristics_groups so ratings embed):
- latham (264: 155 corn + 109 soy) — Latham Hi-Tech Seeds, Alexander IA.
WordPress REST enum (/wp-json/wp/v2/varieties) + /products/<slug>/ detail
HTML. Scale 1-9 LOWER=better (reversed, like NK/AgriPro).
- stine (217: 58 corn + 159 soy) — Stine Seed, Adel IA (largest US
independent). sitemap enum + /{crop}/traits/<slug>/<code>/ detail HTML.
Corn 1-9 (9=best); soy qualitative.
- first_choice (78: 52 corn + 22 soy + 4 wheat) — 1st Choice Seeds,
Rushville IN (employee-owned). Per-crop sitemap -> detail HTML. Scale
0-10 higher=better. ~40 older corn pages thin at source; wheat
identity-only.
- burrus (64: 38 corn + 26 soy) — Burrus Seed, Arenzville IL. Seedware
JSON API. Scale 1-10 (10=best). Brands Burrus/Power Plus/DONMARIO.
robots ai-train=no + named-bot blocks; operator opted in, scraper uses a
non-blacklisted UA + honors Crawl-delay 10.
All 623 validated through rag.chunk.chunks_from_variety (0 errors; 6
identity-only pages from source gaps). No chunk.py change needed (identity
sources auto-route to chunks_from_variety).
Docs:
- sources.json: 4 entries + Hoegemeyer added to _excluded_sources. The
Corteva ToU (shared across pioneer.com / hoegemeyer.com / therightseed.com
/ corteva.com + the Vylor spinoff) bans scrapers + competitive use, so the
whole Corteva family is one excluded ToU domain.
- docs_mcp/lessons.md: rating-scales updated with all 4 directions +
an explicit cross-vendor warning (Latham 1=best vs Stine/Burrus higher=best
— never compare raw numbers without _scale_direction).
- README + CLAUDE corpus inventory: now 2,268 variety + 6,787 trial records.
CI rebuilds the index from the committed corpus.
scrape/
Per-vendor seed catalog scrapers + the runner that dispatches to
them. Each source lives in scrape/sources/<name>.py with a main()
entrypoint. The runner is a thin shim:
python -m scrape.runner --source bayer_seeds --force
python -m scrape.runner --source golden_harvest --limit 20
python -m scrape.runner --all # only GREEN sources
Output layout
Each scraper writes:
corpus/<source>/<source_key>.md— LLM-visible body (chunk_0 preamble + the variety's marketing + agronomic narrative)corpus/<source>/<source_key>.json— sidecar metadata (per CLAUDE.md's canonical schema)
source_key is a stable per-vendor slug — typically <brand>-<sku>
lowercased, e.g. dekalb-dkc62-08rib. Stability matters: it's the
join key the MCP uses for get_page(source, source_key).
Sources
| Source | Module | Verdict | Notes |
|---|---|---|---|
bayer_seeds |
bayer_seeds.py |
🟢 | DEKALB + Asgrow + WestBred, ~475 varieties |
golden_harvest |
golden_harvest.py |
🟢 | ~175 varieties, 9-to-1 disease scale (reverse) |
nk |
nk.py |
🟢 | 29 varieties, ratings in CDN PDFs |
agripro |
agripro.py |
🟢 | 24 wheat varieties |
becks_pfr |
becks_pfr.py |
🟡 | 2,089 research docs via public Sanity GROQ |
becks_products |
becks_products.py |
🟡 | 860 products, identity-only (SeedIQ-gated) |
Pioneer is intentionally absent — see CLAUDE.md and the curated
Pioneer fallback in docs_mcp/lessons.md.
Tips
- Sniff before you scrape. Most catalogs are SPAs that call a
backend API. The recon docs in
~/.claude/projects/-home-justin/ memory/reference_seed_vendor_recon.mdalready capture the endpoints; if you find new ones, update that file. - Idempotent re-scrapes. Without
--force, skip pages already on disk. With--force, re-fetch everything — that's the monthly cron mode. - Respect the portals. Backoff on 429s. Set a recognizable
user-agent (
seed-mcp-scraper/<version>). - Normalize at chunk time, not at scrape time. The chunker
(Phase 2) handles the 9-to-1 → 1-9 disease-scale flip for Golden
Harvest, NOT this scraper. Sidecar JSON should preserve the
vendor's raw values + a
_scale_directionfield; the chunker reads that and normalizes the markdown body.
changelog.py
Reusable as-is from the template. Walks git diff --name-status
output for the commit summary, and git log for the digest history
(Phase 13).