Files

T

Image rebuild (skip scrape) / build (push) Failing after 7s

Details

seed-mcp scaffold: clone docs-mcp-template, customize for crop_seed PRODUCT_NAME

Sibling project to crop-chem-docs, same MCP-template lineage. Corpus is
seed/hybrid varieties across 6 vendors instead of pesticide labels.

What's customized vs. the template:
- CLAUDE.md: vendor matrix, build priority, Pioneer fallback policy,
  canonical sidecar schema (per-crop), Golden Harvest disease-scale
  reversal gotcha, no-IPv6 / HTTPS-clone note
- README.md: vendor coverage table, tool list, phase status
- Dockerfile: PRODUCT_NAME=crop_seed default, sources.json (not
  bundles.json), HYBRID_SEARCH=true, OLLAMA_URL + RERANK_URL Docker
  DNS defaults (same llama-rerank sidecar as crop-chem-docs)
- .gitea/workflows/refresh.yml: monthly cron (seed catalogs move
  slowly), 5 GREEN scraper steps, corpus-YYYY.MM.DD tag for Drawbar
  pinning, continue-on-error on GC step
- .gitea/workflows/image-only.yml: paths filter + cancel-in-progress
  concurrency group
- scripts/registry_gc.py: lifted from crop-chem-docs (correct Gitea
  packages API URL + UA header to bypass CF block on default
  Python-urllib UA)
- sources.json: catalog of 6 sources + scope_filter + per-source
  schema notes + Pioneer-exclusion rationale
- scrape/runner.py: dispatcher with --all = GREEN-only
- scrape/sources/{bayer_seeds,golden_harvest,nk,agripro,becks_pfr,
  becks_products}.py: stub modules with implementation notes
- docs_mcp/server.py: PRODUCT_NAME default → crop_seed,
  PRODUCT_DOCS_URL → repo URL

Pioneer is intentionally NOT a source. ToS bans automation; dealer
locator is login-gated. The MCP returns a curated fallback lesson
directing the user to pioneer.com.

Next phases:
- Phase 1: implement bayer_seeds (lift-and-shift from crop-chem-docs
  Bayer scraper; same __NEXT_DATA__ infra)
- Phase 7: curate eval/queries.jsonl
- Phase 11: lessons.md with Pioneer fallback + disease-scale notes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-25 12:28:49 -04:00

2.4 KiB

Raw Permalink Blame History

scrape/

Per-vendor seed catalog scrapers + the runner that dispatches to them. Each source lives in scrape/sources/<name>.py with a main() entrypoint. The runner is a thin shim:

python -m scrape.runner --source bayer_seeds --force
python -m scrape.runner --source golden_harvest --limit 20
python -m scrape.runner --all                # only GREEN sources

Output layout

Each scraper writes:

corpus/<source>/<source_key>.md — LLM-visible body (chunk_0 preamble + the variety's marketing + agronomic narrative)
corpus/<source>/<source_key>.json — sidecar metadata (per CLAUDE.md's canonical schema)

source_key is a stable per-vendor slug — typically <brand>-<sku> lowercased, e.g. dekalb-dkc62-08rib. Stability matters: it's the join key the MCP uses for get_page(source, source_key).

Sources

Source	Module	Verdict	Notes
`bayer_seeds`	`bayer_seeds.py`	🟢	DEKALB + Asgrow + WestBred, ~475 varieties
`golden_harvest`	`golden_harvest.py`	🟢	~175 varieties, 9-to-1 disease scale (reverse)
`nk`	`nk.py`	🟢	29 varieties, ratings in CDN PDFs
`agripro`	`agripro.py`	🟢	24 wheat varieties
`becks_pfr`	`becks_pfr.py`	🟡	2,089 research docs via public Sanity GROQ
`becks_products`	`becks_products.py`	🟡	860 products, identity-only (SeedIQ-gated)

Pioneer is intentionally absent — see CLAUDE.md and the curated Pioneer fallback in docs_mcp/lessons.md.

Tips

Sniff before you scrape. Most catalogs are SPAs that call a backend API. The recon docs in ~/.claude/projects/-home-justin/ memory/reference_seed_vendor_recon.md already capture the endpoints; if you find new ones, update that file.
Idempotent re-scrapes. Without --force, skip pages already on disk. With --force, re-fetch everything — that's the monthly cron mode.
Respect the portals. Backoff on 429s. Set a recognizable user-agent (seed-mcp-scraper/<version>).
Normalize at chunk time, not at scrape time. The chunker (Phase 2) handles the 9-to-1 → 1-9 disease-scale flip for Golden Harvest, NOT this scraper. Sidecar JSON should preserve the vendor's raw values + a _scale_direction field; the chunker reads that and normalizes the markdown body.

changelog.py

Reusable as-is from the template. Walks git diff --name-status output for the commit summary, and git log for the digest history (Phase 13).

2.4 KiB Raw Permalink Blame History