justin
75f714b454
deploy/docker-compose.yml — replace <product>/<registry> placeholders with concrete values for Drawbar's stack: - image: git.jpaul.io/justin/seed-mcp:latest (CF tunnel for pulls; CI pushes via LAN 192.168.0.2:1234 to avoid 100 MB body cap) - container_name: seed-mcp - port 8001:8000 (8001 host-side to not collide with crop-chem-docs on 8000) - PRODUCT_NAME=crop_seed, hybrid search enabled, stateless HTTP - llama-rerank shared with crop-chem-docs (NOT redefined here — expected to already be in Drawbar's parent compose network) - networks.drawbar-mcp external: true so seed-mcp joins the existing cross-MCP shared network .gitignore — corpus/ is now COMMITTED, not ignored. The monthly refresh workflow scrapes and commits corpus changes; the image-only workflow rebuilds indexes from the committed corpus. Allowing the corpus to flow through git means the :corpus-YYYY.MM.DD image tag pins to a specific seed-catalog snapshot. chroma/ and bm25/ remain ignored — those are deterministically derived from corpus. Initial committed snapshot: 614 varieties. - bayer_seeds: 475 (DEKALB 288 + Asgrow 102 + WestBred 85) - golden_harvest: 139 (Syngenta corn + soy; 36 sitemap URLs 302-redirected = discontinued) rag/chunk.py — normalize brand and crop to uppercase/lowercase in Chroma metadata so cross-vendor brand-filter lookups don't break on casing inconsistency (Bayer stores "DEKALB", Golden Harvest stores "Golden Harvest"; _build_where uppercases user-supplied brand which matched the former but not the latter pre-fix). Sidecar JSON keeps original casing for display. Stub scrapers (nk, agripro, becks_pfr, becks_products) — change return code from 2 to 0 so the monthly-refresh CI workflow doesn't fail on deferred sources. Real implementations will return 0 on success / 1 on failure when they ship. Smoke-tested cross-vendor retrieval against the 614-chunk index: - list_versions shows both vendors with correct facet counts - broad "corn hybrid 100 RM" query returns both DEKALB and Golden Harvest hits in top 5 - brand='Golden Harvest' filter returns 3 GH-only varieties - variety-code prefilter still works (E085Z5 → top hit on GH) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scrape/
Per-vendor seed catalog scrapers + the runner that dispatches to
them. Each source lives in scrape/sources/<name>.py with a main()
entrypoint. The runner is a thin shim:
python -m scrape.runner --source bayer_seeds --force
python -m scrape.runner --source golden_harvest --limit 20
python -m scrape.runner --all # only GREEN sources
Output layout
Each scraper writes:
corpus/<source>/<source_key>.md— LLM-visible body (chunk_0 preamble + the variety's marketing + agronomic narrative)corpus/<source>/<source_key>.json— sidecar metadata (per CLAUDE.md's canonical schema)
source_key is a stable per-vendor slug — typically <brand>-<sku>
lowercased, e.g. dekalb-dkc62-08rib. Stability matters: it's the
join key the MCP uses for get_page(source, source_key).
Sources
| Source | Module | Verdict | Notes |
|---|---|---|---|
bayer_seeds |
bayer_seeds.py |
🟢 | DEKALB + Asgrow + WestBred, ~475 varieties |
golden_harvest |
golden_harvest.py |
🟢 | ~175 varieties, 9-to-1 disease scale (reverse) |
nk |
nk.py |
🟢 | 29 varieties, ratings in CDN PDFs |
agripro |
agripro.py |
🟢 | 24 wheat varieties |
becks_pfr |
becks_pfr.py |
🟡 | 2,089 research docs via public Sanity GROQ |
becks_products |
becks_products.py |
🟡 | 860 products, identity-only (SeedIQ-gated) |
Pioneer is intentionally absent — see CLAUDE.md and the curated
Pioneer fallback in docs_mcp/lessons.md.
Tips
- Sniff before you scrape. Most catalogs are SPAs that call a
backend API. The recon docs in
~/.claude/projects/-home-justin/ memory/reference_seed_vendor_recon.mdalready capture the endpoints; if you find new ones, update that file. - Idempotent re-scrapes. Without
--force, skip pages already on disk. With--force, re-fetch everything — that's the monthly cron mode. - Respect the portals. Backoff on 429s. Set a recognizable
user-agent (
seed-mcp-scraper/<version>). - Normalize at chunk time, not at scrape time. The chunker
(Phase 2) handles the 9-to-1 → 1-9 disease-scale flip for Golden
Harvest, NOT this scraper. Sidecar JSON should preserve the
vendor's raw values + a
_scale_directionfield; the chunker reads that and normalizes the markdown body.
changelog.py
Reusable as-is from the template. Walks git diff --name-status
output for the commit summary, and git log for the digest history
(Phase 13).