justin
30b182e28a
User flagged LG, AgriGold, and Ebbert's (local Ohio breeder) are
all active in farmer territory. Built three scrapers — corpus now
covers 5,839 chunks across 11 brands.
Net new varieties: 310
lg_seeds 170 — corn 78 + soy 63 + alfalfa 16 + sorghum 13
→ adds FIRST alfalfa coverage (FD 3-5 range)
agrigold 111 — corn 60 + soy 51
ebberts_seeds 29 — corn 17 + soy 12 (regional OH/IN breeder)
scrape/sources/lg_seeds.py — embedded-JSON pattern (cleanest):
- /products/<crop> pages have a `var products = [...]` blob with the
variety summary (Variety, Maturity, Traits[], Bullets[], CropType).
- Per-variety detail page (/products/<crop>/<Variety>) carries the
ratings as `<span class="bar-N">` where N is 1-9 on the canonical
scale. Same 9=best direction as Bayer / Golden Harvest.
- Three sections per page: Characteristics / Management / Disease
Tolerance, plus a few qualitative bars ("Tar Spot Susceptible",
"Fungicide Response High") preserved as text values.
scrape/sources/agrigold.py — 5-circle scale:
- Listing page has 60+ /corn/explore-corn-hybrids/<CODE> URLs.
- Detail page renders ratings as <div class="scale"> blocks with 5
child <div class="circle"> elements, of which N have class
"circle selected" → rating N on a 1-5 scale.
- 7 sections per page incl. Silage Characteristics (Dairy Silage
Rating, NDFd 30 Hr, Crude Protein), Planting Applications, Soil
Adaptability, Plant Characteristics, Product Features.
- Distinct rating direction (1-5 vs Bayer's 1-9) — declared in
_scale_direction so chunker preamble renders correctly.
scrape/sources/ebberts_seeds.py — small regional breeder, verbatim
text approach:
- Single page per crop (corn / soybeans / wheat). Each variety is an
<h1> + multi-section CSS-grid block where labels and values are in
separate adjacent cells. Reconstructing perfectly-aligned columns
for a 29-variety total isn't worth the engineering — chunk body
carries the verbatim text in document order, LLM can read the
tabular content.
- Scale: 1-5 (1 = best, lower = more resistant), inferred from
marketing-vs-rating cross-checks ("Robust tall plants" + STANDABILITY
1.0 → 1 = best).
- Politeness: robots.txt asks for Crawl-delay: 5; honored.
All three new scrapers smoke-tested:
- LG corn LG5701 RM 116 SmartStax → 3 characteristic groups with
Disease Tolerance ratings (Northern/Southern Leaf Blight 8-9, etc.)
- AgriGold A616-30 RM 86 VT2RIB → 7 groups incl. silage and soil
adaptability ratings
- Ebbert's 7000TR RIB RM 100 → 1098-char verbatim body covering
CHARACTERISTICS, DISEASE RATINGS, herbicide tolerance, etc.
Corpus state after this PR:
- 5,839 chunks (was 5,529)
- 11 brands (was 8)
- 8 crops (corn 3047, soy 2209, silage 359, wheat 123, sorghum 49,
cotton 30, alfalfa 16, canola 6) — alfalfa is brand-new
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scrape/
Per-vendor seed catalog scrapers + the runner that dispatches to
them. Each source lives in scrape/sources/<name>.py with a main()
entrypoint. The runner is a thin shim:
python -m scrape.runner --source bayer_seeds --force
python -m scrape.runner --source golden_harvest --limit 20
python -m scrape.runner --all # only GREEN sources
Output layout
Each scraper writes:
corpus/<source>/<source_key>.md— LLM-visible body (chunk_0 preamble + the variety's marketing + agronomic narrative)corpus/<source>/<source_key>.json— sidecar metadata (per CLAUDE.md's canonical schema)
source_key is a stable per-vendor slug — typically <brand>-<sku>
lowercased, e.g. dekalb-dkc62-08rib. Stability matters: it's the
join key the MCP uses for get_page(source, source_key).
Sources
| Source | Module | Verdict | Notes |
|---|---|---|---|
bayer_seeds |
bayer_seeds.py |
🟢 | DEKALB + Asgrow + WestBred, ~475 varieties |
golden_harvest |
golden_harvest.py |
🟢 | ~175 varieties, 9-to-1 disease scale (reverse) |
nk |
nk.py |
🟢 | 29 varieties, ratings in CDN PDFs |
agripro |
agripro.py |
🟢 | 24 wheat varieties |
becks_pfr |
becks_pfr.py |
🟡 | 2,089 research docs via public Sanity GROQ |
becks_products |
becks_products.py |
🟡 | 860 products, identity-only (SeedIQ-gated) |
Pioneer is intentionally absent — see CLAUDE.md and the curated
Pioneer fallback in docs_mcp/lessons.md.
Tips
- Sniff before you scrape. Most catalogs are SPAs that call a
backend API. The recon docs in
~/.claude/projects/-home-justin/ memory/reference_seed_vendor_recon.mdalready capture the endpoints; if you find new ones, update that file. - Idempotent re-scrapes. Without
--force, skip pages already on disk. With--force, re-fetch everything — that's the monthly cron mode. - Respect the portals. Backoff on 429s. Set a recognizable
user-agent (
seed-mcp-scraper/<version>). - Normalize at chunk time, not at scrape time. The chunker
(Phase 2) handles the 9-to-1 → 1-9 disease-scale flip for Golden
Harvest, NOT this scraper. Sidecar JSON should preserve the
vendor's raw values + a
_scale_directionfield; the chunker reads that and normalizes the markdown body.
changelog.py
Reusable as-is from the template. Walks git diff --name-status
output for the commit summary, and git log for the digest history
(Phase 13).