Phase 2/3: chunker + indexer + MCP server tools
Phase 2 — Chunking and indexing
- rag/chunk.py: replace template chunker with seed-variety-specific
chunks_from_variety(). One chunk per variety (varieties are small
and named-rating retrieval signal is best kept together). Output
is rebuilt deterministically from the sidecar JSON: every value is
verbatim from the source, only framing language ("Disease ratings
(1-9, 9=best):") is template glue. Anti-hallucination contract:
same sidecar in → same chunk out, never a fabricated rating.
Metadata flattened to Chroma-safe primitives (str/int/float/bool):
source, source_key, vendor, brand, crop, product_name,
product_id, source_url, rm (corn), mg (soy), wheat_class,
release_year, trait_codes_csv, rating_scale.
- rag/index.py: walks corpus/<source>/<source_key>.json sidecars
via the new chunker. Default PRODUCT_NAME=crop_seed so the
Chroma collection is crop_seed_docs.
- rag/bm25.py: filterable columns updated to seed-domain facets
(source/vendor/brand/crop/source_key) instead of the template's
version/platform/product.
Phase 3 — MCP server tools wired up
- search_docs: hybrid dense (Chroma) + BM25 (FTS5) retrieval with
RRF fusion. Optional filters: crop, brand, vendor, source.
Variety-code prefilter pins exact source_key / product_name /
hybrid_prefix matches at the top — dense embeddings have no
semantic neighbor for tokens like "DKC62-08RIB" and RRF can let
noise float to #1 without this pin. Each response carries the
variety's source URL inline so the agent can cite.
- get_page(source, source_key): emits a structured ratings header
(verbatim from sidecar, table per characteristics group, vendor
positioning, regional listings) followed by the raw indexed body.
This is the canonical fact-check surface.
- list_versions(): facet discovery — distinct sources, vendors,
brands, crops across the corpus.
- lookup_variety(source_key, source?): returns the raw sidecar JSON
for one variety. The agent should call this BEFORE quoting any
specific rating value to a farmer — guaranteed verbatim.
Smoke tests against 475 indexed Bayer varieties:
- list_versions returns 475 varieties, 1 source, 1 vendor, 3 brands,
3 crops with correct per-brand counts (288/102/85).
- Semantic ag queries find the right candidates: short-season
drought-tolerant corn → DKC44-97RIB at RM 94 (in 90-95 band);
SCN+MG3 soybean → Asgrow XF varieties with explicit SCN R3 ratings;
Phytophthora Rps3a soy → AG07XF4 (right gene); stripe-rust
wheat → WestBred WB1376CLP (Yellow Rust 2 = best).
- Variety-code lookups work via prefilter: DKC62-08RIB, AG29XF4,
WB6430 all return as #1 hit. BM25 confirms ranking unambiguously
(top-1 score -13.2 vs -8.5 for #2 on "DKC62-08RIB ratings").
- Server boots cleanly in stdio AND streamable-http modes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+52
-99
@@ -1,17 +1,14 @@
|
||||
"""SQLite FTS5-backed BM25 retrieval over the same chunks Chroma indexes.
|
||||
|
||||
Hybrid retrieval (BM25 + dense + Reciprocal Rank Fusion) addresses a
|
||||
limit of single-tower dense embeddings: when a query has specific
|
||||
technical terms (filenames, language names, error codes, API paths),
|
||||
the dense embedding doesn't bridge from the query into a short
|
||||
code-focused chunk. The chunk loses to the much larger crowd of
|
||||
prose chunks that semantically match the query topic.
|
||||
|
||||
BM25 handles this directly. Lexical overlap on rare terms ("python",
|
||||
"create_vpg.py", "PROTECTED_SITE_ID", "applyUpgrade") scores those
|
||||
chunks high. Fused with the dense ranking via RRF, the hybrid result
|
||||
is strictly better than either alone for the queries we've seen
|
||||
fail.
|
||||
single-tower dense embedding's weakness on rare technical tokens —
|
||||
for seed-mcp that's variety codes ("DKC62-08RIB"), trait codes
|
||||
("XF", "VT2PRIB"), disease abbreviations ("SCN", "SDS", "Goss's"),
|
||||
and Rps gene names ("Rps1c", "Rps3a"). Dense embeddings don't bridge
|
||||
queries like "Rps3a soybean" cleanly into the relevant chunk; BM25
|
||||
matches them directly. Fused with the dense ranking via RRF, the
|
||||
hybrid result is strictly better than either alone for the queries
|
||||
we expect from the farm-advisor agent.
|
||||
|
||||
Why SQLite FTS5:
|
||||
- In the stdlib. Zero new deps.
|
||||
@@ -19,36 +16,13 @@ Why SQLite FTS5:
|
||||
`rag.index --rebuild` regenerates from corpus.
|
||||
- Built-in `bm25()` ranking function. No knobs to tune that matter
|
||||
for our use case (k1=1.2, b=0.75 defaults are fine).
|
||||
- Builds 70k+ chunks in seconds. Faster than the Chroma rebuild's
|
||||
embedding step by 100×, so it adds basically nothing to the
|
||||
full-rebuild cycle.
|
||||
- Builds <1k chunks in milliseconds; adds nothing to rebuild time.
|
||||
|
||||
Schema is two tables to keep filtering clean. FTS5 doesn't filter
|
||||
nicely on its own columns; the content_rowid pattern keeps an
|
||||
external metadata table joinable by rowid:
|
||||
|
||||
CREATE TABLE chunks_meta (
|
||||
rowid INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
id TEXT UNIQUE,
|
||||
bundle_id TEXT, page_id TEXT, version TEXT,
|
||||
platform TEXT, product TEXT, ordinal INTEGER
|
||||
);
|
||||
CREATE VIRTUAL TABLE chunks_fts USING fts5(
|
||||
text,
|
||||
tokenize = 'porter unicode61 remove_diacritics 2',
|
||||
content = 'chunks_meta',
|
||||
content_rowid = 'rowid'
|
||||
);
|
||||
|
||||
Queries:
|
||||
|
||||
SELECT m.id, bm25(chunks_fts) AS score
|
||||
FROM chunks_meta m
|
||||
JOIN chunks_fts f ON m.rowid = f.rowid
|
||||
WHERE f MATCH ?
|
||||
AND m.version = ? -- optional metadata filter
|
||||
ORDER BY bm25(chunks_fts) -- lower = better in FTS5
|
||||
LIMIT ?;
|
||||
external metadata table joinable by rowid. For seed-mcp the
|
||||
filterable columns are seed-domain facets — source, vendor, brand,
|
||||
crop, source_key — rather than the docs-template version/platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -63,42 +37,30 @@ log = logging.getLogger(__name__)
|
||||
# Default location: bm25/<product>_docs.db at the repo root, next to chroma/.
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DEFAULT_DB_DIR = ROOT / "bm25"
|
||||
DEFAULT_DB_NAME = "<product>_docs.db"
|
||||
DEFAULT_DB_NAME = "crop_seed_docs.db"
|
||||
|
||||
# Columns we expose as filterable metadata. Mirrors what _build_where in
|
||||
# docs_mcp/server.py accepts so the same filter dicts work for both
|
||||
# Chroma and BM25 without per-retriever translation in the caller.
|
||||
FILTER_COLUMNS = ("bundle_id", "page_id", "version", "platform", "product", "ordinal")
|
||||
# Columns we expose as filterable metadata. Mirrors what
|
||||
# ``docs_mcp.server._build_where`` accepts so the same filter dict
|
||||
# works for both Chroma and BM25 without per-retriever translation.
|
||||
FILTER_COLUMNS = ("source", "vendor", "brand", "crop", "source_key", "ordinal")
|
||||
|
||||
|
||||
# Allowlist tokenizer for free-text queries. FTS5's parser chokes on lots
|
||||
# of punctuation we routinely see in user queries (".10.9", "?", "VPG's",
|
||||
# em-dash, etc.). Rather than blocklist every operator, just keep
|
||||
# alphanumerics + a few separators and replace everything else with a
|
||||
# space. This loses the ability to phrase-search ("exact match") but we
|
||||
# don't expose that to users anyway — they ask natural-language questions
|
||||
# and want the answer, not a Boolean DSL.
|
||||
# Allowlist tokenizer for free-text queries. FTS5's parser chokes on
|
||||
# lots of punctuation we routinely see in farmer queries ("Rps1c",
|
||||
# "SCN-resistant", "0.05 MG", em-dashes). Rather than blocklist every
|
||||
# operator, keep alphanumerics + a few separators and replace
|
||||
# everything else with a space.
|
||||
_KEEP_RE = re.compile(r"[^A-Za-z0-9_\s]")
|
||||
# FTS5 reserves these Boolean operator KEYWORDS at the token level —
|
||||
# stripping them avoids accidental phrase-query behavior when a user
|
||||
# query happens to contain bare "AND", "OR", "NOT", "NEAR".
|
||||
# FTS5 reserves these Boolean operator KEYWORDS at the token level.
|
||||
_BOOLEAN_KW_RE = re.compile(r"(?<!\w)(AND|OR|NOT|NEAR)(?!\w)")
|
||||
|
||||
|
||||
def _sanitize_query(text: str) -> str:
|
||||
"""Reduce a natural-language query to an FTS5 OR-of-tokens query.
|
||||
|
||||
Two transformations:
|
||||
|
||||
1. Non-alphanumeric → space (drops punctuation; "10.9?" becomes
|
||||
"10 9"). Lets us handle versions, parens, question marks, etc.
|
||||
without inviting FTS5 parse errors.
|
||||
2. Boolean keywords stripped (FTS5 reserves AND/OR/NOT/NEAR).
|
||||
3. Tokens explicitly OR'd. FTS5's default is AND-of-tokens — for
|
||||
any non-trivial natural-language query that means zero hits
|
||||
(no chunk contains every word). OR semantics is what we want:
|
||||
BM25 already weights documents containing more query terms
|
||||
higher, so we don't lose precision, but we DO gain recall.
|
||||
See ``crop-chem-docs`` for the rationale; same transformation
|
||||
applies here. OR semantics maximizes recall — BM25 already
|
||||
weights documents with more query-term matches higher.
|
||||
"""
|
||||
cleaned = _KEEP_RE.sub(" ", text)
|
||||
cleaned = _BOOLEAN_KW_RE.sub(" ", cleaned)
|
||||
@@ -113,9 +75,9 @@ def _where_to_sql(where: dict | None) -> tuple[str, list[Any]]:
|
||||
|
||||
Accepts the same shapes ``docs_mcp.server._build_where`` produces:
|
||||
|
||||
None → ("", [])
|
||||
{"version": "10.9"} → ("AND m.version = ?", ["10.9"])
|
||||
{"$and": [{...}, {...}]} → ("AND m.X = ? AND m.Y = ?", [...])
|
||||
None → ("", [])
|
||||
{"crop": "corn"} → ("AND m.crop = ?", ["corn"])
|
||||
{"$and": [{...}, {...}]} → ("AND m.X = ? AND m.Y = ?", [...])
|
||||
|
||||
Unknown keys are silently dropped (defensive — better to over-match
|
||||
than to crash on a filter we don't know).
|
||||
@@ -158,35 +120,32 @@ class BM25Index:
|
||||
def build(self, records: list[dict]) -> int:
|
||||
"""Rebuild the index from scratch from `records`.
|
||||
|
||||
`records` is the same list ``rag.index.page_records`` produces:
|
||||
`records` is the same list ``rag.index.variety_records`` produces:
|
||||
``[{"id": ..., "text": ..., "metadata": {...}}, ...]``. Bulk
|
||||
insert wrapped in a transaction — single-digit seconds for the
|
||||
full 73k-chunk corpus.
|
||||
insert wrapped in a transaction.
|
||||
"""
|
||||
self.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# Drop and recreate. Idempotent rebuild.
|
||||
if self.db_path.exists():
|
||||
self.db_path.unlink()
|
||||
with sqlite3.connect(self.db_path) as con:
|
||||
con.executescript(self._schema_sql())
|
||||
con.executemany(
|
||||
"INSERT INTO chunks_meta (id, bundle_id, page_id, version, "
|
||||
"platform, product, ordinal) VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
"INSERT INTO chunks_meta "
|
||||
"(id, source, vendor, brand, crop, source_key, ordinal) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
[
|
||||
(
|
||||
r["id"],
|
||||
r["metadata"].get("bundle_id") or "",
|
||||
r["metadata"].get("page_id") or "",
|
||||
r["metadata"].get("version") or "",
|
||||
r["metadata"].get("platform") or "",
|
||||
r["metadata"].get("product") or "",
|
||||
r["metadata"].get("source") or "",
|
||||
r["metadata"].get("vendor") or "",
|
||||
r["metadata"].get("brand") or "",
|
||||
r["metadata"].get("crop") or "",
|
||||
r["metadata"].get("source_key") or "",
|
||||
int(r["metadata"].get("ordinal") or 0),
|
||||
)
|
||||
for r in records
|
||||
],
|
||||
)
|
||||
# Populate the FTS5 contentless-ish table by rowid. We populated
|
||||
# chunks_meta first; rowids align with insertion order.
|
||||
con.executemany(
|
||||
"INSERT INTO chunks_fts (rowid, text) VALUES (?, ?)",
|
||||
[
|
||||
@@ -210,15 +169,12 @@ class BM25Index:
|
||||
|
||||
FTS5's bm25() returns NEGATIVE numbers — more relevant docs have
|
||||
smaller (more negative) scores. We order ASC so the first row is
|
||||
the most relevant. Callers that need a "rank" should enumerate
|
||||
the returned list.
|
||||
the most relevant.
|
||||
"""
|
||||
sanitized = _sanitize_query(text)
|
||||
if not sanitized:
|
||||
return []
|
||||
where_sql, params = _where_to_sql(where)
|
||||
# FTS5 MATCH wants the unaliased table name on its left, so we use
|
||||
# chunks_fts (no alias) and JOIN by rowid against chunks_meta.
|
||||
sql = (
|
||||
"SELECT m.id, bm25(chunks_fts) AS score "
|
||||
"FROM chunks_fts "
|
||||
@@ -232,17 +188,13 @@ class BM25Index:
|
||||
cur = con.execute(sql, [sanitized, *params, n])
|
||||
return [(row[0], float(row[1])) for row in cur.fetchall()]
|
||||
except sqlite3.OperationalError as e:
|
||||
# FTS5 syntax error (rare after sanitization) or db missing.
|
||||
# Caller decides whether to fall back to dense-only.
|
||||
log.warning("bm25 query failed (%s); query=%r", e, sanitized[:80])
|
||||
return []
|
||||
|
||||
def exists(self) -> bool:
|
||||
"""Cheap probe — does the index file exist on disk?"""
|
||||
return self.db_path.exists()
|
||||
|
||||
def count(self) -> int:
|
||||
"""Number of chunks indexed. 0 if the db is missing or empty."""
|
||||
if not self.exists():
|
||||
return 0
|
||||
try:
|
||||
@@ -257,18 +209,19 @@ class BM25Index:
|
||||
def _schema_sql() -> str:
|
||||
return """
|
||||
CREATE TABLE chunks_meta (
|
||||
rowid INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
id TEXT UNIQUE NOT NULL,
|
||||
bundle_id TEXT,
|
||||
page_id TEXT,
|
||||
version TEXT,
|
||||
platform TEXT,
|
||||
product TEXT,
|
||||
ordinal INTEGER
|
||||
rowid INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
id TEXT UNIQUE NOT NULL,
|
||||
source TEXT,
|
||||
vendor TEXT,
|
||||
brand TEXT,
|
||||
crop TEXT,
|
||||
source_key TEXT,
|
||||
ordinal INTEGER
|
||||
);
|
||||
CREATE INDEX idx_meta_version ON chunks_meta(version);
|
||||
CREATE INDEX idx_meta_platform ON chunks_meta(platform);
|
||||
CREATE INDEX idx_meta_bundle ON chunks_meta(bundle_id);
|
||||
CREATE INDEX idx_meta_source ON chunks_meta(source);
|
||||
CREATE INDEX idx_meta_crop ON chunks_meta(crop);
|
||||
CREATE INDEX idx_meta_brand ON chunks_meta(brand);
|
||||
CREATE INDEX idx_meta_source_key ON chunks_meta(source_key);
|
||||
|
||||
CREATE VIRTUAL TABLE chunks_fts USING fts5(
|
||||
text,
|
||||
|
||||
+298
-100
@@ -1,126 +1,324 @@
|
||||
"""Markdown chunker — paragraph-aware, ~400-600 token target.
|
||||
"""Chunker for seed-variety corpus.
|
||||
|
||||
Adjust the chunking strategy per product if your page format differs
|
||||
significantly from prose. The output shape (id, text, metadata) is
|
||||
fixed by the downstream Chroma + BM25 indexing in rag/index.py — don't
|
||||
change that.
|
||||
Each variety becomes ONE chunk by default. Variety pages are small
|
||||
(typically 2-3 KB of useful signal) and nomic-embed-text handles up
|
||||
to ~8 K tokens cleanly. Splitting a variety across chunks dilutes
|
||||
the named-rating embeddings (e.g. "SCN resistance 7") that farmers
|
||||
search by — keep them together.
|
||||
|
||||
The key knob you'll tune per product is chunk-0. Dense retrieval lands
|
||||
on chunk 0 first for most queries. Make it a synthetic chunk built
|
||||
from:
|
||||
The chunk text is a synthetic preamble assembled deterministically
|
||||
from the sidecar JSON. Every value in the chunk text comes verbatim
|
||||
from the source. The framing words ("Disease ratings (1-9, 9=best):",
|
||||
"Maturity group:", etc.) are template glue — *we add structure, we
|
||||
do NOT add facts*. Given the same sidecar, this chunker always
|
||||
produces the same chunk text. That's the anti-hallucination
|
||||
contract: the retriever can never surface a rating value that
|
||||
wasn't in the source.
|
||||
|
||||
- the page title (as natural-language H1)
|
||||
- a 1-sentence task description (you'll have to generate this — for
|
||||
pages that already have a "## Overview" or "## Introduction" the
|
||||
first sentence usually works)
|
||||
- a keyword bag of important terms (filenames, API names, error
|
||||
codes — the rare technical tokens that BM25 lights up on)
|
||||
Metadata is flattened to Chroma-safe primitives (str/int/float/bool):
|
||||
|
||||
Without a rich chunk 0, dense retrieval gets dominated by the much
|
||||
larger prose body, and short pages (script examples, reference cards)
|
||||
get buried.
|
||||
source "bayer_seeds"
|
||||
source_key "dekalb-dkc075-70rib"
|
||||
vendor "Bayer"
|
||||
brand "DEKALB"
|
||||
crop "corn" | "soybeans" | "wheat"
|
||||
product_name "DKC075-70RIB BRAND BLEND"
|
||||
product_id canonical full id
|
||||
source_url the variety's page URL
|
||||
rm corn RM as int when parseable (else absent)
|
||||
mg soy MG as float when parseable (else absent)
|
||||
release_year int when known
|
||||
trait_codes_csv comma-separated trait codes for substring search
|
||||
rating_scale "1-9 (9 = best)" — chunker should ALWAYS attach
|
||||
this so downstream code can detect a flip
|
||||
ordinal chunk index within variety (0-based)
|
||||
|
||||
Lists like ``regional_recommendations`` and the full per-rating dicts
|
||||
do NOT fit Chroma's metadata constraints — they stay in the sidecar
|
||||
JSON, surfaced by ``get_page`` / ``lookup_variety``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Iterator
|
||||
|
||||
|
||||
# Approximate token estimate from char count. Tunable — set per
|
||||
# embedder if the default 4 chars/token is wrong.
|
||||
CHARS_PER_TOKEN = 4
|
||||
TARGET_TOKENS = 500
|
||||
TARGET_CHARS = TARGET_TOKENS * CHARS_PER_TOKEN
|
||||
# Rating-group classification. The source publishes characteristics
|
||||
# grouped by label; we map those labels to one of three buckets in
|
||||
# the chunk preamble so retrieval gets coherent text. Group labels not
|
||||
# listed here fall into "other" and are still emitted, just in their
|
||||
# own section.
|
||||
DISEASE_GROUP_LABELS = {
|
||||
"DISEASE RATINGS",
|
||||
"PEST AND DISEASE RESISTANCE",
|
||||
}
|
||||
AGRONOMIC_GROUP_LABELS = {
|
||||
"GROWTH",
|
||||
"HARVEST",
|
||||
"PRODUCTION",
|
||||
"KEY CHARACTERISTICS",
|
||||
"QUALITY",
|
||||
}
|
||||
MANAGEMENT_GROUP_LABELS = {
|
||||
"MANAGEMENT",
|
||||
"HERBICIDE",
|
||||
"SENSITIVITY",
|
||||
"PLANT DESCRIPTION",
|
||||
}
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
return max(1, len(text) // CHARS_PER_TOKEN)
|
||||
def _parse_rm(value: object) -> int | None:
|
||||
"""Best-effort RM-days int. Returns None if not a clean integer
|
||||
(e.g. wheat's qualitative 'Early'/'Medium-Early' values)."""
|
||||
if value is None:
|
||||
return None
|
||||
s = str(value).strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
# Handle floats stored as strings ("105.0") and the trailing
|
||||
# tenths sometimes seen on early corn ("75").
|
||||
return int(float(s))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def split_paragraphs(md: str) -> list[str]:
|
||||
"""Split markdown into paragraph-ish blocks.
|
||||
def _parse_mg(value: object) -> float | None:
|
||||
"""Best-effort MG float. Soy MGs go from 00 to 9.0 with one decimal."""
|
||||
if value is None:
|
||||
return None
|
||||
s = str(value).strip()
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return float(s)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
Keeps fenced code blocks together (don't slice through ```).
|
||||
Headings start new paragraphs.
|
||||
|
||||
def _format_items(items: list[dict]) -> str:
|
||||
"""Render `[{characteristic, value}, ...]` to a compact inline list."""
|
||||
out: list[str] = []
|
||||
for it in items:
|
||||
ch = (it.get("characteristic") or "").strip()
|
||||
v = (it.get("value") or "").strip()
|
||||
if ch and v:
|
||||
out.append(f"{ch} {v}")
|
||||
elif ch:
|
||||
out.append(f"{ch} —")
|
||||
return ", ".join(out)
|
||||
|
||||
|
||||
def _render_variety_chunk(sidecar: dict) -> str:
|
||||
"""Build the dense preamble for one variety from its sidecar JSON.
|
||||
|
||||
Faithful to source: every numeric/categorical *value* is verbatim
|
||||
from ``sidecar``. The only generated text is the framing language.
|
||||
"""
|
||||
blocks: list[str] = []
|
||||
current: list[str] = []
|
||||
in_fence = False
|
||||
for line in md.splitlines(keepends=True):
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("```"):
|
||||
in_fence = not in_fence
|
||||
current.append(line)
|
||||
lines: list[str] = []
|
||||
|
||||
# ---- Identity line --------------------------------------------------
|
||||
name = sidecar.get("product_name") or sidecar.get("source_key") or ""
|
||||
brand = (sidecar.get("brand") or "").strip()
|
||||
vendor = sidecar.get("vendor") or ""
|
||||
crop = (sidecar.get("crop") or "").strip()
|
||||
crop_label = crop.capitalize() if crop else ""
|
||||
ident = f"# {name}"
|
||||
sub = " ".join(filter(None, [
|
||||
f"({brand.title()} {crop_label} variety, {vendor})" if brand and crop_label and vendor else "",
|
||||
]))
|
||||
lines.append(ident)
|
||||
if sub:
|
||||
lines.append("")
|
||||
lines.append(sub)
|
||||
|
||||
# ---- Identity body --------------------------------------------------
|
||||
facts: list[str] = []
|
||||
|
||||
rm = sidecar.get("relative_maturity")
|
||||
mg = sidecar.get("maturity_group")
|
||||
wc = sidecar.get("wheat_class")
|
||||
if crop == "corn" and rm:
|
||||
facts.append(f"Relative maturity {rm}")
|
||||
elif crop == "soybeans" and mg:
|
||||
facts.append(f"Maturity group {mg}")
|
||||
elif crop == "wheat":
|
||||
if rm:
|
||||
facts.append(f"Maturity {rm}")
|
||||
if wc:
|
||||
facts.append(f"Wheat class {wc}")
|
||||
|
||||
traits = sidecar.get("trait_stack") or []
|
||||
trait_descs = sidecar.get("trait_descriptions") or []
|
||||
if traits:
|
||||
if trait_descs:
|
||||
facts.append(
|
||||
"Trait stack: "
|
||||
+ ", ".join(traits)
|
||||
+ " ("
|
||||
+ "; ".join(trait_descs)
|
||||
+ ")"
|
||||
)
|
||||
else:
|
||||
facts.append("Trait stack: " + ", ".join(traits))
|
||||
|
||||
if sidecar.get("release_year"):
|
||||
facts.append(f"Released {sidecar['release_year']}")
|
||||
|
||||
if facts:
|
||||
lines.append("")
|
||||
lines.append(". ".join(facts) + ".")
|
||||
|
||||
# ---- Positioning ----------------------------------------------------
|
||||
pos = (sidecar.get("positioning_statement") or "").strip()
|
||||
if pos:
|
||||
lines.append("")
|
||||
lines.append(f"Positioning: {pos}")
|
||||
|
||||
# ---- Ratings, bucketed for retrieval --------------------------------
|
||||
scale = sidecar.get("_scale_direction") or "(scale direction not declared)"
|
||||
groups = sidecar.get("characteristics_groups") or []
|
||||
disease: list[dict] = []
|
||||
agronomic: list[dict] = []
|
||||
management: list[dict] = []
|
||||
other: list[tuple[str, list[dict]]] = []
|
||||
for g in groups:
|
||||
label = (g.get("label") or "").upper().strip()
|
||||
items = g.get("items") or []
|
||||
if not items:
|
||||
continue
|
||||
if in_fence:
|
||||
current.append(line)
|
||||
continue
|
||||
if stripped.startswith("#"):
|
||||
if current:
|
||||
blocks.append("".join(current).strip())
|
||||
current = []
|
||||
current.append(line)
|
||||
continue
|
||||
if not stripped and current and not "".join(current).strip().endswith("\n\n"):
|
||||
current.append(line)
|
||||
blocks.append("".join(current).strip())
|
||||
current = []
|
||||
continue
|
||||
current.append(line)
|
||||
if current:
|
||||
blocks.append("".join(current).strip())
|
||||
return [b for b in blocks if b]
|
||||
if label in DISEASE_GROUP_LABELS:
|
||||
disease.extend(items)
|
||||
elif label in AGRONOMIC_GROUP_LABELS:
|
||||
agronomic.extend(items)
|
||||
elif label in MANAGEMENT_GROUP_LABELS:
|
||||
management.extend(items)
|
||||
else:
|
||||
other.append((g.get("label") or "Other", items))
|
||||
|
||||
if disease:
|
||||
lines.append("")
|
||||
lines.append(f"Disease ratings ({scale}): {_format_items(disease)}.")
|
||||
if agronomic:
|
||||
lines.append("")
|
||||
lines.append(f"Agronomic ratings ({scale}): {_format_items(agronomic)}.")
|
||||
if management:
|
||||
lines.append("")
|
||||
lines.append(f"Management notes: {_format_items(management)}.")
|
||||
for label, items in other:
|
||||
lines.append("")
|
||||
lines.append(f"{label.title()}: {_format_items(items)}.")
|
||||
|
||||
# ---- Strengths narrative --------------------------------------------
|
||||
strengths = sidecar.get("strengths") or []
|
||||
if strengths:
|
||||
lines.append("")
|
||||
lines.append("Strengths and management notes:")
|
||||
for s in strengths:
|
||||
s = (s or "").strip()
|
||||
if s:
|
||||
lines.append(f"- {s}")
|
||||
|
||||
# ---- Regional listings (compact, not the agronomist emails) ---------
|
||||
rec = sidecar.get("regional_recommendations") or []
|
||||
if rec:
|
||||
names = sorted({
|
||||
(r.get("product_list_name") or "").strip()
|
||||
for r in rec
|
||||
if (r.get("product_list_name") or "").strip()
|
||||
})
|
||||
if names:
|
||||
lines.append("")
|
||||
lines.append("Listed in regional seed guides: " + "; ".join(names) + ".")
|
||||
|
||||
# ---- Provenance footer (must always be in the chunk text so it
|
||||
# can never be lost between retrieval and LLM rendering) --------
|
||||
urls = sidecar.get("source_urls") or []
|
||||
if urls:
|
||||
lines.append("")
|
||||
lines.append(f"Source: {urls[0]}")
|
||||
|
||||
return "\n".join(lines).strip() + "\n"
|
||||
|
||||
|
||||
def chunks_from_page(
|
||||
text: str,
|
||||
page_id: str,
|
||||
metadata: dict,
|
||||
def _flat_metadata(sidecar: dict) -> dict:
|
||||
"""Distil sidecar into Chroma-safe metadata (primitives only)."""
|
||||
md: dict = {
|
||||
"source": sidecar.get("source") or "",
|
||||
"source_key": sidecar.get("source_key") or "",
|
||||
"vendor": sidecar.get("vendor") or "",
|
||||
"brand": sidecar.get("brand") or "",
|
||||
"crop": sidecar.get("crop") or "",
|
||||
"product_name": sidecar.get("product_name") or "",
|
||||
"product_id": sidecar.get("product_id") or "",
|
||||
"source_url": (sidecar.get("source_urls") or [""])[0],
|
||||
"rating_scale": sidecar.get("_scale_direction") or "",
|
||||
}
|
||||
rm = _parse_rm(sidecar.get("relative_maturity"))
|
||||
mg = _parse_mg(sidecar.get("maturity_group"))
|
||||
if rm is not None:
|
||||
md["rm"] = rm
|
||||
if mg is not None:
|
||||
md["mg"] = mg
|
||||
ry = sidecar.get("release_year")
|
||||
if isinstance(ry, int):
|
||||
md["release_year"] = ry
|
||||
traits = sidecar.get("trait_stack") or []
|
||||
if traits:
|
||||
# Comma-delimited for partial-match / human eyeballing.
|
||||
# Bracket-padded so `LIKE '%,XF,%'` finds whole tokens.
|
||||
md["trait_codes_csv"] = "," + ",".join(traits) + ","
|
||||
if sidecar.get("wheat_class"):
|
||||
md["wheat_class"] = sidecar["wheat_class"]
|
||||
return md
|
||||
|
||||
|
||||
def chunks_from_variety(
|
||||
sidecar_path: Path | str,
|
||||
*,
|
||||
md_path: Path | str | None = None,
|
||||
) -> Iterator[dict]:
|
||||
"""Yield chunk dicts ready for index.py to upsert.
|
||||
"""Yield chunk dict(s) for one variety. Currently emits exactly one.
|
||||
|
||||
The synthetic chunk 0 is the per-product customization point. The
|
||||
default below is a simple title + body-first-paragraph; rewrite
|
||||
for richer retrieval signal (see module docstring).
|
||||
Args:
|
||||
sidecar_path: path to the variety's JSON sidecar.
|
||||
md_path: ignored (the chunker rebuilds from sidecar); kept
|
||||
in the signature in case a future split-chunker
|
||||
wants the rendered body.
|
||||
"""
|
||||
paragraphs = split_paragraphs(text)
|
||||
if not paragraphs:
|
||||
return
|
||||
|
||||
# ----- Chunk 0: synthetic anchor for dense retrieval ---------
|
||||
title = metadata.get("title") or page_id
|
||||
first_para = next((p for p in paragraphs if not p.startswith("#")), "")
|
||||
chunk0_body = (
|
||||
f"# {title}\n\n"
|
||||
f"{first_para[:300]}"
|
||||
# TODO per product: append a keyword bag here (filenames,
|
||||
# API names, error codes) for BM25 + dense joint coverage.
|
||||
)
|
||||
sidecar = json.loads(Path(sidecar_path).read_text(encoding="utf-8"))
|
||||
text = _render_variety_chunk(sidecar)
|
||||
meta = _flat_metadata(sidecar)
|
||||
chunk_id = f"{meta['source']}::{meta['source_key']}::0"
|
||||
yield {
|
||||
"id": f"{metadata['bundle_id']}::{page_id}::0",
|
||||
"text": chunk0_body,
|
||||
"metadata": {**metadata, "ordinal": 0},
|
||||
"id": chunk_id,
|
||||
"text": text,
|
||||
"metadata": {**meta, "ordinal": 0},
|
||||
}
|
||||
|
||||
# ----- Body chunks: pack paragraphs up to TARGET_CHARS -------
|
||||
ordinal = 1
|
||||
buf: list[str] = []
|
||||
buf_chars = 0
|
||||
for p in paragraphs:
|
||||
if buf_chars + len(p) > TARGET_CHARS and buf:
|
||||
yield {
|
||||
"id": f"{metadata['bundle_id']}::{page_id}::{ordinal}",
|
||||
"text": "\n\n".join(buf),
|
||||
"metadata": {**metadata, "ordinal": ordinal},
|
||||
}
|
||||
ordinal += 1
|
||||
buf = []
|
||||
buf_chars = 0
|
||||
buf.append(p)
|
||||
buf_chars += len(p)
|
||||
if buf:
|
||||
yield {
|
||||
"id": f"{metadata['bundle_id']}::{page_id}::{ordinal}",
|
||||
"text": "\n\n".join(buf),
|
||||
"metadata": {**metadata, "ordinal": ordinal},
|
||||
}
|
||||
|
||||
# ----- Backwards-compat shim for the template's index.py -------------------
|
||||
#
|
||||
# The template's ``rag.index.page_records`` calls
|
||||
# ``chunks_from_page(md, page_id, base_meta)`` which doesn't know about
|
||||
# sidecar JSON. We accept that signature but ignore it — index.py has
|
||||
# been updated to use ``chunks_from_variety`` directly, and this shim
|
||||
# is here only so a stray import of the old name doesn't break.
|
||||
#
|
||||
def chunks_from_page(text: str, page_id: str, metadata: dict) -> Iterator[dict]:
|
||||
"""Deprecated for seed-mcp; prefer ``chunks_from_variety``."""
|
||||
# Best-effort: if metadata carries a sidecar_path, dispatch.
|
||||
sidecar_path = metadata.get("_sidecar_path")
|
||||
if sidecar_path:
|
||||
yield from chunks_from_variety(sidecar_path)
|
||||
return
|
||||
# Fallback — emit a single chunk of the raw markdown with whatever
|
||||
# metadata we have. Better than crashing if someone calls this.
|
||||
chunk_id = f"{metadata.get('source','unknown')}::{page_id}::0"
|
||||
yield {
|
||||
"id": chunk_id,
|
||||
"text": text,
|
||||
"metadata": {**metadata, "ordinal": 0},
|
||||
}
|
||||
|
||||
+25
-38
@@ -1,15 +1,19 @@
|
||||
"""Build Chroma (and optionally BM25) indexes from corpus on disk.
|
||||
"""Build Chroma (and BM25) indexes from the seed corpus on disk.
|
||||
|
||||
Reads `corpus/<bundle>/<page>.{md,json}`, chunks each page, upserts
|
||||
into Chroma. With --rebuild, drops + recreates the collection (clean
|
||||
state). With --bm25-only, skips Chroma and rebuilds only the FTS5
|
||||
index — useful for fast iteration when chunking didn't change.
|
||||
Reads ``corpus/<source>/<source_key>.json`` sidecars, chunks each
|
||||
variety via ``rag.chunk.chunks_from_variety``, upserts into Chroma.
|
||||
With ``--rebuild``, drops + recreates the collection (clean state).
|
||||
With ``--bm25-only``, skips Chroma and rebuilds only the FTS5 index
|
||||
— useful for fast iteration when the chunker didn't change.
|
||||
|
||||
Collection name is ``<PRODUCT_NAME>_docs`` (default: ``crop_seed_docs``).
|
||||
Override via the PRODUCT_NAME env var.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Iterator
|
||||
@@ -17,7 +21,7 @@ from typing import Iterator
|
||||
import chromadb
|
||||
from chromadb.config import Settings
|
||||
|
||||
from .chunk import chunks_from_page
|
||||
from .chunk import chunks_from_variety
|
||||
from .embeddings import embedding_function
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -27,39 +31,21 @@ ROOT = Path(__file__).resolve().parent.parent
|
||||
CORPUS = ROOT / "corpus"
|
||||
CHROMA_DIR = ROOT / "chroma"
|
||||
|
||||
# Collection name — convention: <product>_docs. Override via env if needed.
|
||||
import os
|
||||
PRODUCT_NAME = os.environ.get("PRODUCT_NAME", "myproduct")
|
||||
PRODUCT_NAME = os.environ.get("PRODUCT_NAME", "crop_seed")
|
||||
COLLECTION = f"{PRODUCT_NAME}_docs"
|
||||
|
||||
|
||||
def page_records() -> Iterator[dict]:
|
||||
"""Walk corpus/, yield chunks for every page."""
|
||||
def variety_records() -> Iterator[dict]:
|
||||
"""Walk ``corpus/<source>/<source_key>.json``, yield one chunk per
|
||||
variety."""
|
||||
if not CORPUS.exists():
|
||||
log.error("corpus/ doesn't exist; run the scraper first")
|
||||
log.error("corpus/ doesn't exist; run a scraper first")
|
||||
return
|
||||
for bundle_dir in sorted(CORPUS.iterdir()):
|
||||
if not bundle_dir.is_dir() or bundle_dir.name.startswith("."):
|
||||
for source_dir in sorted(CORPUS.iterdir()):
|
||||
if not source_dir.is_dir() or source_dir.name.startswith("."):
|
||||
continue
|
||||
for md_path in sorted(bundle_dir.glob("*.md")):
|
||||
page_id = md_path.stem
|
||||
sidecar = md_path.with_suffix(".json")
|
||||
if not sidecar.exists():
|
||||
log.warning("skipping %s — no JSON sidecar", md_path)
|
||||
continue
|
||||
md = md_path.read_text()
|
||||
meta = json.loads(sidecar.read_text())
|
||||
# Surface common filter fields at the chunk-metadata level
|
||||
# so Chroma's `where` filter can use them.
|
||||
base_meta = {
|
||||
"bundle_id": bundle_dir.name,
|
||||
"page_id": page_id,
|
||||
"title": meta.get("title") or "",
|
||||
"version": meta.get("version") or "",
|
||||
"platform": meta.get("platform") or "",
|
||||
"product": meta.get("product") or "",
|
||||
}
|
||||
yield from chunks_from_page(md, page_id, base_meta)
|
||||
for sidecar_path in sorted(source_dir.glob("*.json")):
|
||||
yield from chunks_from_variety(sidecar_path)
|
||||
|
||||
|
||||
def upsert_to_chroma(records: list[dict]) -> int:
|
||||
@@ -67,7 +53,7 @@ def upsert_to_chroma(records: list[dict]) -> int:
|
||||
path=str(CHROMA_DIR),
|
||||
settings=Settings(anonymized_telemetry=False),
|
||||
)
|
||||
# Drop + recreate for --rebuild semantics
|
||||
# Drop + recreate for --rebuild semantics.
|
||||
try:
|
||||
client.delete_collection(COLLECTION)
|
||||
except Exception:
|
||||
@@ -101,8 +87,11 @@ def main() -> int:
|
||||
|
||||
log.info("reading corpus from %s", CORPUS)
|
||||
t0 = time.time()
|
||||
records = list(page_records())
|
||||
records = list(variety_records())
|
||||
log.info("loaded %d chunks in %.1fs", len(records), time.time() - t0)
|
||||
if not records:
|
||||
log.error("no chunks — is corpus/ populated?")
|
||||
return 1
|
||||
|
||||
if args.bm25_only:
|
||||
from .bm25 import BM25Index
|
||||
@@ -118,8 +107,6 @@ def main() -> int:
|
||||
n = upsert_to_chroma(records)
|
||||
log.info("chroma: %d chunks in %.1fs", n, time.time() - t_c)
|
||||
|
||||
# Build BM25 too — see PLAN.md Phase 8. Safe to remove this block
|
||||
# for products that don't need hybrid retrieval.
|
||||
try:
|
||||
from .bm25 import BM25Index
|
||||
t_b = time.time()
|
||||
|
||||
Reference in New Issue
Block a user