justin
75f714b454
deploy/docker-compose.yml — replace <product>/<registry> placeholders with concrete values for Drawbar's stack: - image: git.jpaul.io/justin/seed-mcp:latest (CF tunnel for pulls; CI pushes via LAN 192.168.0.2:1234 to avoid 100 MB body cap) - container_name: seed-mcp - port 8001:8000 (8001 host-side to not collide with crop-chem-docs on 8000) - PRODUCT_NAME=crop_seed, hybrid search enabled, stateless HTTP - llama-rerank shared with crop-chem-docs (NOT redefined here — expected to already be in Drawbar's parent compose network) - networks.drawbar-mcp external: true so seed-mcp joins the existing cross-MCP shared network .gitignore — corpus/ is now COMMITTED, not ignored. The monthly refresh workflow scrapes and commits corpus changes; the image-only workflow rebuilds indexes from the committed corpus. Allowing the corpus to flow through git means the :corpus-YYYY.MM.DD image tag pins to a specific seed-catalog snapshot. chroma/ and bm25/ remain ignored — those are deterministically derived from corpus. Initial committed snapshot: 614 varieties. - bayer_seeds: 475 (DEKALB 288 + Asgrow 102 + WestBred 85) - golden_harvest: 139 (Syngenta corn + soy; 36 sitemap URLs 302-redirected = discontinued) rag/chunk.py — normalize brand and crop to uppercase/lowercase in Chroma metadata so cross-vendor brand-filter lookups don't break on casing inconsistency (Bayer stores "DEKALB", Golden Harvest stores "Golden Harvest"; _build_where uppercases user-supplied brand which matched the former but not the latter pre-fix). Sidecar JSON keeps original casing for display. Stub scrapers (nk, agripro, becks_pfr, becks_products) — change return code from 2 to 0 so the monthly-refresh CI workflow doesn't fail on deferred sources. Real implementations will return 0 on success / 1 on failure when they ship. Smoke-tested cross-vendor retrieval against the 614-chunk index: - list_versions shows both vendors with correct facet counts - broad "corn hybrid 100 RM" query returns both DEKALB and Golden Harvest hits in top 5 - brand='Golden Harvest' filter returns 3 GH-only varieties - variety-code prefilter still works (E085Z5 → top hit on GH) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
48 lines
1.5 KiB
Python
48 lines
1.5 KiB
Python
"""Beck's PFR (Practical Farm Research) scraper.
|
|
|
|
Source: Public Sanity GROQ API at ``https://mc8v24rf.api.sanity.io``.
|
|
No authentication required — Beck's exposes their CMS content store
|
|
publicly. ~2,089 documents going back to 2015.
|
|
|
|
Sanity query endpoint:
|
|
``/v1/data/query/production?query=<groq>``
|
|
|
|
Useful GROQ for PFR docs (the projectId / dataset are public):
|
|
|
|
*[_type == "pfrStudy"] {
|
|
_id, title, year, crop, slug, summary, body, attachments
|
|
}
|
|
|
|
Records are research studies, not variety identity — head-to-head
|
|
yield trials, fungicide timing, planting-date studies, hybrid-by-
|
|
population, biological seed treatments, etc.
|
|
|
|
Treat differently from variety scrapers:
|
|
- One record per study, not per variety
|
|
- chunk_0 preamble includes the study's tl;dr finding (extract from
|
|
the ``summary`` field if present, or first paragraph of ``body``)
|
|
- Crop tag (corn/soy/wheat) for filtering
|
|
- Year tag — older PFR studies are still relevant but search should
|
|
let the user weight recency
|
|
|
|
Polite rate limit: Sanity is generous but no auth means we should
|
|
keep concurrency ≤4 and pause ~250ms between batches.
|
|
|
|
TODO: implement.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
print("becks_pfr: deferred — public Sanity GROQ at mc8v24rf.api.sanity.io, ~2089 research docs",
|
|
file=sys.stderr)
|
|
# Return 0 so the monthly CI workflow doesn't fail when this
|
|
# source is listed but not yet implemented.
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main(sys.argv[1:]))
|