claude
54094a0d43
Independent third-party performance data — land-grant programs that test every entered brand side-by-side with replication + LSD stats. This is the legitimate way to get Pioneer / DEKALB / Brevant / Channel performance the corpus can't scrape directly (data_type=trial, results[] shape; falls through the trial chunker). - illinois_vt_trials (30 docs, 1,392 rows) — U of Illinois VT. Per-region XLSX (openpyxl), corn + soy + WHEAT, 2024+2025. Rich per-site agronomic metadata; corn-following-corn vs -soybean kept distinct. - iowa_icpt_trials (24 docs, 674 rows) — Iowa State ICPT. ASP.NET GridView (viewstate postback for year/district), corn + soy by district x season. - ohio_ocpt_trials (69 docs, 4,647 rows) — OSU/CFAES OCPT. Report PDF (pdfplumber; per-site column groups split by header Yield-token count + x-coord footnote bucketing), corn + soy per site, 2024+2025. 91 distinct seed brands across the three; majors confirmed present in the independent rankings: DEKALB 395, Golden Harvest 249, Channel 241, NK 212, Xitavo 135, LG 103, Pioneer 88, Asgrow 59. (A brand only appears where it ENTERED a given program — e.g. Brevant not in Iowa, DEKALB/Channel not in Illinois — true negatives, not parse gaps.) - rag/chunk.py: gated `include_region` on _render_gh_plot_chunk; the 3 university sources route through it so the region/district is in the embedded chunk + labeled "variety trial (cross-vendor, independent third-party)". Existing plot sources (gh/lg/agrigold/proharvest) unchanged. - requirements.txt: openpyxl (Illinois XLSX; scrape-time only). - sources.json + README/CLAUDE/lessons: registered + attributed; lessons trial-data + Pioneer entries updated (Pioneer/DEKALB performance now available indirectly via these trials). Validation: all 123 chunk via rag.chunk.chunks_from_trial (0 errors), 0 out-of-range yields, 0 dup keys. Public land-grant data; attribution recorded in each tos_note. CI rebuilds the index from the committed corpus.
scrape/
Per-vendor seed catalog scrapers + the runner that dispatches to
them. Each source lives in scrape/sources/<name>.py with a main()
entrypoint. The runner is a thin shim:
python -m scrape.runner --source bayer_seeds --force
python -m scrape.runner --source golden_harvest --limit 20
python -m scrape.runner --all # only GREEN sources
Output layout
Each scraper writes:
corpus/<source>/<source_key>.md— LLM-visible body (chunk_0 preamble + the variety's marketing + agronomic narrative)corpus/<source>/<source_key>.json— sidecar metadata (per CLAUDE.md's canonical schema)
source_key is a stable per-vendor slug — typically <brand>-<sku>
lowercased, e.g. dekalb-dkc62-08rib. Stability matters: it's the
join key the MCP uses for get_page(source, source_key).
Sources
| Source | Module | Verdict | Notes |
|---|---|---|---|
bayer_seeds |
bayer_seeds.py |
🟢 | DEKALB + Asgrow + WestBred, ~475 varieties |
golden_harvest |
golden_harvest.py |
🟢 | ~175 varieties, 9-to-1 disease scale (reverse) |
nk |
nk.py |
🟢 | 29 varieties, ratings in CDN PDFs |
agripro |
agripro.py |
🟢 | 24 wheat varieties |
becks_pfr |
becks_pfr.py |
🟡 | 2,089 research docs via public Sanity GROQ |
becks_products |
becks_products.py |
🟡 | 860 products, identity-only (SeedIQ-gated) |
Pioneer is intentionally absent — see CLAUDE.md and the curated
Pioneer fallback in docs_mcp/lessons.md.
Tips
- Sniff before you scrape. Most catalogs are SPAs that call a
backend API. The recon docs in
~/.claude/projects/-home-justin/ memory/reference_seed_vendor_recon.mdalready capture the endpoints; if you find new ones, update that file. - Idempotent re-scrapes. Without
--force, skip pages already on disk. With--force, re-fetch everything — that's the monthly cron mode. - Respect the portals. Backoff on 429s. Set a recognizable
user-agent (
seed-mcp-scraper/<version>). - Normalize at chunk time, not at scrape time. The chunker
(Phase 2) handles the 9-to-1 → 1-9 disease-scale flip for Golden
Harvest, NOT this scraper. Sidecar JSON should preserve the
vendor's raw values + a
_scale_directionfield; the chunker reads that and normalizes the markdown body.
changelog.py
Reusable as-is from the template. Walks git diff --name-status
output for the commit summary, and git log for the digest history
(Phase 13).