justin
335c33465b
## Phase 7 — Eval harness
eval/retrievers.py + rag/retrieval.py: Retriever protocol with
DenseRetriever, BM25Retriever, HybridRetriever (RRF k=60),
RerankedRetriever (llama.cpp /v1/rerank). retrievers.py is now a
thin shim re-exporting from rag.retrieval so the MCP server can
use the same code at request time without making eval/ a runtime
dep.
eval/run_eval.py: drives N retrievers against eval/queries.jsonl,
computes MRR / Recall@K / nDCG@K, emits a markdown report with a
summary table + per-query breakdown for the first retriever. Each
query carries expected (source, source_key) tuples — matches the
labels-domain page-level keying.
eval/queries.jsonl: 35 curated queries — 25 brand-name (Warrant,
Huskie, Roundup Custom, Liberty, Authority, Headline, Trivapro,
Poncho, Lorsban, Sencor, Acuron, ...) + 10 intent/semantic
("what controls horseweed before soybean", "fungicide for fusarium
head blight", "rainfast interval for glyphosate", ...).
## Phase 8 — Hybrid retrieval (BM25 + dense + RRF)
docs_mcp/server.py: search_docs now branches on HYBRID_SEARCH env.
When on, _search_chunks runs both Chroma + BM25 (rag/bm25.py
existing impl), fuses on chunk_id with reciprocal-rank-fusion
(RRF k=60), and returns the combined pool. Dense-only path
unchanged when HYBRID_SEARCH is unset. The rendering layer
(_format_hit) is untouched.
The RERANK_URL hook is also wired (_rerank_pool sends docs to
llama.cpp /v1/rerank, truncated to 2000 chars per the jina-reranker
n_ctx_train=1024 batch-rejection gotcha). Fails open to base order
on any exception.
## Baseline numbers (k=5, pool=50, 35 queries)
| Retriever | MRR | Recall@5 | nDCG@5 |
|------------|-------|----------|--------|
| dense | 0.027 | 0.086 | 0.041 |
| bm25 | 0.544 | 0.586 | 0.524 |
| hybrid-rrf | 0.114 | 0.114 | 0.108 |
Headline: BM25 dominates because farmers search for products by
brand name, and brand names are exact-match tokens that lexical
search nails. Dense is poor — semantic embeddings spread across
similar products and don't preferentially weight brand-name tokens.
Textbook RRF hurts when one retriever is much weaker than the
other: dense's irrelevant top-50 pollute the fused pool with
ties at 1/(60+rank). Phase 6 reranker is the planned fix —
the reranker scores each (query, chunk) pair independently
and can recover the right answer regardless of base order.
Per-query report at eval/results/baseline.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
36 lines
6.6 KiB
JSON
36 lines
6.6 KiB
JSON
{"query": "Warrant herbicide rate for soybean", "expected": [{"source": "bayer", "source_key": "warrant"}, {"source": "epa_ppls", "source_key": "524-591"}], "tags": ["brand", "herbicide", "soybean"]}
|
|
{"query": "Huskie wheat herbicide tank mix", "expected": [{"source": "bayer", "source_key": "huskie"}, {"source": "bayer", "source_key": "huskie-complete"}], "tags": ["brand", "herbicide", "wheat"]}
|
|
{"query": "Harness 20G granular corn herbicide", "expected": [{"source": "bayer", "source_key": "harness"}, {"source": "epa_ppls", "source_key": "524-487"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Laudis tembotrione post-emergence corn", "expected": [{"source": "bayer", "source_key": "laudis"}, {"source": "epa_ppls", "source_key": "264-860"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Roundup Custom glyphosate burndown application rate", "expected": [{"source": "epa_ppls", "source_key": "524-677"}, {"source": "epa_ppls", "source_key": "524-475"}], "tags": ["brand", "herbicide", "glyphosate"]}
|
|
{"query": "Liberty 280 SL glufosinate ammonium soybean", "expected": [{"source": "epa_ppls", "source_key": "7969-448"}], "tags": ["brand", "herbicide", "soybean"]}
|
|
{"query": "Atrazine 4L corn pre-emergence rate per acre", "expected": [{"source": "epa_ppls", "source_key": "5905-7877"}], "tags": ["active-ingredient", "herbicide", "corn", "atrazine"]}
|
|
{"query": "Albaugh dicamba DMA salt application restrictions", "expected": [{"source": "epa_ppls", "source_key": "42750-40"}], "tags": ["active-ingredient", "herbicide", "dicamba"]}
|
|
{"query": "Authority 4F sulfentrazone soybean residual", "expected": [{"source": "epa_ppls", "source_key": "279-3146"}], "tags": ["brand", "herbicide", "soybean"]}
|
|
{"query": "Prowl 10-G pendimethalin granular pre-plant", "expected": [{"source": "epa_ppls", "source_key": "241-254"}], "tags": ["brand", "herbicide"]}
|
|
{"query": "Callisto GT mesotrione corn postemergence broadleaf control", "expected": [{"source": "epa_ppls", "source_key": "100-1470"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Acuron Flexi corn pre-emergence S-metolachlor", "expected": [{"source": "epa_ppls", "source_key": "100-1568"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Sencor 4 flowable metribuzin soybean waterhemp", "expected": [{"source": "epa_ppls", "source_key": "264-735"}], "tags": ["brand", "herbicide", "soybean", "waterhemp"]}
|
|
{"query": "Broadstrike trifluralin pre-plant incorporated", "expected": [{"source": "epa_ppls", "source_key": "62719-222"}], "tags": ["brand", "herbicide"]}
|
|
{"query": "Headline azoxystrobin pyraclostrobin wheat foliar fungicide", "expected": [{"source": "epa_ppls", "source_key": "7969-186"}], "tags": ["brand", "fungicide", "wheat"]}
|
|
{"query": "Trivapro pydiflumetofen corn fungicide tar spot", "expected": [{"source": "epa_ppls", "source_key": "100-1613"}], "tags": ["brand", "fungicide", "corn"]}
|
|
{"query": "Poncho 600 clothianidin seed treatment corn", "expected": [{"source": "epa_ppls", "source_key": "7969-458"}], "tags": ["brand", "insecticide", "seed-treatment"]}
|
|
{"query": "Gustafson Lorsban 30 chlorpyrifos granular corn rootworm", "expected": [{"source": "epa_ppls", "source_key": "264-932"}], "tags": ["brand", "insecticide", "corn"]}
|
|
{"query": "RT-3 glyphosate potassium salt herbicide", "expected": [{"source": "bayer", "source_key": "rt-3"}], "tags": ["brand", "herbicide"]}
|
|
{"query": "Roundup PowerMAX 3 glyphosate K-salt rate", "expected": [{"source": "bayer", "source_key": "roundup-powermax-3"}, {"source": "epa_ppls", "source_key": "524-659"}], "tags": ["brand", "herbicide"]}
|
|
{"query": "Nortron SC ethofumesate sugar beet", "expected": [{"source": "bayer", "source_key": "nortron-sc"}], "tags": ["brand", "herbicide"]}
|
|
{"query": "DiFlexx Duo tembotrione dicamba corn", "expected": [{"source": "bayer", "source_key": "diflexx-duo"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Corvus thiencarbazone-methyl isoxaflutole corn pre-emergence", "expected": [{"source": "bayer", "source_key": "corvus"}, {"source": "epa_ppls", "source_key": "264-1066"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Capreno tembotrione thiencarbazone corn herbicide", "expected": [{"source": "bayer", "source_key": "capreno"}, {"source": "epa_ppls", "source_key": "264-1063"}], "tags": ["brand", "herbicide", "corn"]}
|
|
{"query": "Tilt propiconazole wheat fungicide rust", "expected": [{"source": "epa_ppls", "source_key": "100-617"}], "tags": ["brand", "fungicide", "wheat"]}
|
|
{"query": "what controls horseweed marestail before planting soybean", "expected": [{"source": "epa_ppls", "source_key": "524-475"}, {"source": "epa_ppls", "source_key": "524-677"}], "tags": ["intent", "herbicide", "soybean"]}
|
|
{"query": "what can I tank mix with 2,4-D for burndown in spring", "expected": [{"source": "epa_ppls", "source_key": "5905-7877"}, {"source": "epa_ppls", "source_key": "228-666"}], "tags": ["intent", "herbicide", "burndown"]}
|
|
{"query": "best fungicide for corn tar spot foliar application", "expected": [{"source": "epa_ppls", "source_key": "100-1613"}, {"source": "epa_ppls", "source_key": "100-1547"}], "tags": ["intent", "fungicide", "corn"]}
|
|
{"query": "seed treatment to control wireworm in corn", "expected": [{"source": "epa_ppls", "source_key": "7969-458"}, {"source": "epa_ppls", "source_key": "7969-459"}], "tags": ["intent", "insecticide", "seed-treatment", "corn"]}
|
|
{"query": "pre-emergence residual herbicide for soybean for waterhemp", "expected": [{"source": "epa_ppls", "source_key": "279-3146"}, {"source": "epa_ppls", "source_key": "264-735"}], "tags": ["intent", "herbicide", "soybean", "waterhemp"]}
|
|
{"query": "what insecticide for soybean aphid foliar", "expected": [{"source": "epa_ppls", "source_key": "279-3206"}, {"source": "epa_ppls", "source_key": "264-840"}], "tags": ["intent", "insecticide", "soybean", "aphid"]}
|
|
{"query": "what is the rainfast interval for glyphosate", "expected": [{"source": "epa_ppls", "source_key": "524-475"}, {"source": "epa_ppls", "source_key": "524-677"}], "tags": ["intent", "herbicide", "glyphosate"]}
|
|
{"query": "wheat fungicide for fusarium head blight", "expected": [{"source": "epa_ppls", "source_key": "7969-186"}, {"source": "epa_ppls", "source_key": "100-1547"}], "tags": ["intent", "fungicide", "wheat"]}
|
|
{"query": "endangered species act precautions for pesticide application", "expected": [{"source": "epa_ppls", "source_key": "524-475"}, {"source": "epa_ppls", "source_key": "524-591"}], "tags": ["intent", "regulatory"]}
|
|
{"query": "what herbicide do I use for postemergence broadleaf in corn", "expected": [{"source": "bayer", "source_key": "laudis"}, {"source": "bayer", "source_key": "capreno"}, {"source": "bayer", "source_key": "diflexx-duo"}], "tags": ["intent", "herbicide", "corn"]}
|