Phase 2/3: chunker + indexer + MCP server tools

Phase 2 — Chunking and indexing
- rag/chunk.py: replace template chunker with seed-variety-specific
  chunks_from_variety(). One chunk per variety (varieties are small
  and named-rating retrieval signal is best kept together). Output
  is rebuilt deterministically from the sidecar JSON: every value is
  verbatim from the source, only framing language ("Disease ratings
  (1-9, 9=best):") is template glue. Anti-hallucination contract:
  same sidecar in → same chunk out, never a fabricated rating.
  Metadata flattened to Chroma-safe primitives (str/int/float/bool):
  source, source_key, vendor, brand, crop, product_name,
  product_id, source_url, rm (corn), mg (soy), wheat_class,
  release_year, trait_codes_csv, rating_scale.
- rag/index.py: walks corpus/<source>/<source_key>.json sidecars
  via the new chunker. Default PRODUCT_NAME=crop_seed so the
  Chroma collection is crop_seed_docs.
- rag/bm25.py: filterable columns updated to seed-domain facets
  (source/vendor/brand/crop/source_key) instead of the template's
  version/platform/product.

Phase 3 — MCP server tools wired up
- search_docs: hybrid dense (Chroma) + BM25 (FTS5) retrieval with
  RRF fusion. Optional filters: crop, brand, vendor, source.
  Variety-code prefilter pins exact source_key / product_name /
  hybrid_prefix matches at the top — dense embeddings have no
  semantic neighbor for tokens like "DKC62-08RIB" and RRF can let
  noise float to #1 without this pin. Each response carries the
  variety's source URL inline so the agent can cite.
- get_page(source, source_key): emits a structured ratings header
  (verbatim from sidecar, table per characteristics group, vendor
  positioning, regional listings) followed by the raw indexed body.
  This is the canonical fact-check surface.
- list_versions(): facet discovery — distinct sources, vendors,
  brands, crops across the corpus.
- lookup_variety(source_key, source?): returns the raw sidecar JSON
  for one variety. The agent should call this BEFORE quoting any
  specific rating value to a farmer — guaranteed verbatim.

Smoke tests against 475 indexed Bayer varieties:
- list_versions returns 475 varieties, 1 source, 1 vendor, 3 brands,
  3 crops with correct per-brand counts (288/102/85).
- Semantic ag queries find the right candidates: short-season
  drought-tolerant corn → DKC44-97RIB at RM 94 (in 90-95 band);
  SCN+MG3 soybean → Asgrow XF varieties with explicit SCN R3 ratings;
  Phytophthora Rps3a soy → AG07XF4 (right gene); stripe-rust
  wheat → WestBred WB1376CLP (Yellow Rust 2 = best).
- Variety-code lookups work via prefilter: DKC62-08RIB, AG29XF4,
  WB6430 all return as #1 hit. BM25 confirms ranking unambiguously
  (top-1 score -13.2 vs -8.5 for #2 on "DKC62-08RIB ratings").
- Server boots cleanly in stdio AND streamable-http modes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

rag/index.py

@@ -1,15 +1,19 @@
-"""Build Chroma (and optionally BM25) indexes from corpus on disk.
+"""Build Chroma (and BM25) indexes from the seed corpus on disk.
 
-Reads `corpus/<bundle>/<page>.{md,json}`, chunks each page, upserts
-into Chroma. With --rebuild, drops + recreates the collection (clean
-state). With --bm25-only, skips Chroma and rebuilds only the FTS5
-index — useful for fast iteration when chunking didn't change.
+Reads ``corpus/<source>/<source_key>.json`` sidecars, chunks each
+variety via ``rag.chunk.chunks_from_variety``, upserts into Chroma.
+With ``--rebuild``, drops + recreates the collection (clean state).
+With ``--bm25-only``, skips Chroma and rebuilds only the FTS5 index
+— useful for fast iteration when the chunker didn't change.
+
+Collection name is ``<PRODUCT_NAME>_docs`` (default: ``crop_seed_docs``).
+Override via the PRODUCT_NAME env var.
 """
 from __future__ import annotations
 
 import argparse
-import json
 import logging
+import os
 import time
 from pathlib import Path
 from typing import Iterator
@@ -17,7 +21,7 @@ from typing import Iterator
 import chromadb
 from chromadb.config import Settings
 
-from .chunk import chunks_from_page
+from .chunk import chunks_from_variety
 from .embeddings import embedding_function
 
 log = logging.getLogger(__name__)
@@ -27,39 +31,21 @@ ROOT = Path(__file__).resolve().parent.parent
 CORPUS = ROOT / "corpus"
 CHROMA_DIR = ROOT / "chroma"
 
-# Collection name — convention: <product>_docs. Override via env if needed.
-import os
-PRODUCT_NAME = os.environ.get("PRODUCT_NAME", "myproduct")
+PRODUCT_NAME = os.environ.get("PRODUCT_NAME", "crop_seed")
 COLLECTION = f"{PRODUCT_NAME}_docs"
 
 
-def page_records() -> Iterator[dict]:
-    """Walk corpus/, yield chunks for every page."""
+def variety_records() -> Iterator[dict]:
+    """Walk ``corpus/<source>/<source_key>.json``, yield one chunk per
+    variety."""
     if not CORPUS.exists():
-        log.error("corpus/ doesn't exist; run the scraper first")
+        log.error("corpus/ doesn't exist; run a scraper first")
         return
-    for bundle_dir in sorted(CORPUS.iterdir()):
-        if not bundle_dir.is_dir() or bundle_dir.name.startswith("."):
+    for source_dir in sorted(CORPUS.iterdir()):
+        if not source_dir.is_dir() or source_dir.name.startswith("."):
             continue
-        for md_path in sorted(bundle_dir.glob("*.md")):
-            page_id = md_path.stem
-            sidecar = md_path.with_suffix(".json")
-            if not sidecar.exists():
-                log.warning("skipping %s — no JSON sidecar", md_path)
-                continue
-            md = md_path.read_text()
-            meta = json.loads(sidecar.read_text())
-            # Surface common filter fields at the chunk-metadata level
-            # so Chroma's `where` filter can use them.
-            base_meta = {
-                "bundle_id": bundle_dir.name,
-                "page_id":   page_id,
-                "title":     meta.get("title") or "",
-                "version":   meta.get("version") or "",
-                "platform":  meta.get("platform") or "",
-                "product":   meta.get("product") or "",
-            }
-            yield from chunks_from_page(md, page_id, base_meta)
+        for sidecar_path in sorted(source_dir.glob("*.json")):
+            yield from chunks_from_variety(sidecar_path)
 
 
 def upsert_to_chroma(records: list[dict]) -> int:
@@ -67,7 +53,7 @@ def upsert_to_chroma(records: list[dict]) -> int:
         path=str(CHROMA_DIR),
         settings=Settings(anonymized_telemetry=False),
     )
-    # Drop + recreate for --rebuild semantics
+    # Drop + recreate for --rebuild semantics.
     try:
         client.delete_collection(COLLECTION)
     except Exception:
@@ -101,8 +87,11 @@ def main() -> int:
 
     log.info("reading corpus from %s", CORPUS)
     t0 = time.time()
-    records = list(page_records())
+    records = list(variety_records())
     log.info("loaded %d chunks in %.1fs", len(records), time.time() - t0)
+    if not records:
+        log.error("no chunks — is corpus/ populated?")
+        return 1
 
     if args.bm25_only:
         from .bm25 import BM25Index
@@ -118,8 +107,6 @@ def main() -> int:
     n = upsert_to_chroma(records)
     log.info("chroma: %d chunks in %.1fs", n, time.time() - t_c)
 
-    # Build BM25 too — see PLAN.md Phase 8. Safe to remove this block
-    # for products that don't need hybrid retrieval.
     try:
         from .bm25 import BM25Index
         t_b = time.time()