trial data: workflow scrape steps + lessons.md trial-data guide

.gitea/workflows/refresh.yml — add scrape steps for the new trial
sources (agripro_trials, gh_plot_reports) so the monthly cron
refreshes them alongside the variety sources. gh_plot_reports
is the heaviest single source (~4,600 docs @ 1 req/sec ≈ 70 min);
runs late so an earlier failure doesn't waste time before failing.
Commit-message variable count expanded to surface the trial counts.

docs_mcp/lessons.md — new "trial-data" section telling the agent:

- The two surfaces (search_docs = identity, search_trials = perf)
  are complementary; how to route a farmer question to each.
- What's indexed (GH plot reports cross-vendor, AgriPro regional
  PDFs) vs what's not (Bayer per-variety trials, NK yield results,
  Pioneer, university extension trials).
- Recommended workflow: search_trials → identify top performers →
  lookup_variety on each to verify identity → don't fabricate.
- How to read a GH plot report (per-column headers vary by crop:
  corn/soy use Yield/MST/Test Weight, silage uses Ton/Acre +
  Milk + Beef columns).
- Single-data-point caveat: one plot is one cooperator's field;
  look across multiple plots for a robust recommendation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-25 15:22:08 -04:00
parent 17260c32c8
commit 84b49d8360
2 changed files with 89 additions and 3 deletions
+16 -3
View File
@@ -83,10 +83,21 @@ jobs:
if: ${{ inputs.sources == '' || contains(inputs.sources, 'agripro') }}
run: python -m scrape.runner --source agripro --force
- name: Scrape AgriPro regional trial PDFs
if: ${{ inputs.sources == '' || contains(inputs.sources, 'agripro_trials') }}
run: python -m scrape.runner --source agripro_trials --force
- name: Scrape Golden Harvest plot reports (cross-vendor yield trials)
if: ${{ inputs.sources == '' || contains(inputs.sources, 'gh_plot_reports') }}
# Heaviest single source — ~4,600 docs at 1 req/sec ≈ 70 min.
# Wraps the bulk of CI time; runs late so an earlier failure
# doesn't waste 70 min before failing.
run: python -m scrape.runner --source gh_plot_reports --force
- name: Scrape Beck's PFR research corpus
if: ${{ inputs.sources == '' || contains(inputs.sources, 'becks_pfr') }}
# Heaviest source — ~2,089 docs via public Sanity GROQ.
# No auth, but rate-limit ourselves to be polite.
# Deferred (returns 0 cleanly from a stub) — implementation
# pending. Public Sanity GROQ at mc8v24rf.api.sanity.io.
run: python -m scrape.runner --source becks_pfr --force
# ---- Commit corpus changes + retry-on-race -----------------
@@ -107,8 +118,10 @@ jobs:
n_gh=$(find corpus/golden_harvest -name '*.json' 2>/dev/null | wc -l)
n_nk=$(find corpus/nk -name '*.json' 2>/dev/null | wc -l)
n_ag=$(find corpus/agripro -name '*.json' 2>/dev/null | wc -l)
n_agt=$(find corpus/agripro_trials -name '*.json' 2>/dev/null | wc -l)
n_ghpr=$(find corpus/gh_plot_reports -name '*.json' 2>/dev/null | wc -l)
n_pfr=$(find corpus/becks_pfr -name '*.json' 2>/dev/null | wc -l)
git commit -m "monthly refresh: ${ts} — bayer=${n_bayer} gh=${n_gh} nk=${n_nk} agripro=${n_ag} pfr=${n_pfr}"
git commit -m "monthly refresh: ${ts} — bayer=${n_bayer} gh=${n_gh} nk=${n_nk} agripro=${n_ag} ag_trials=${n_agt} gh_plot_reports=${n_ghpr} pfr=${n_pfr}"
attempt=1
while [ $attempt -le 3 ]; do
if git push; then