trial data: workflow scrape steps + lessons.md trial-data guide
.gitea/workflows/refresh.yml — add scrape steps for the new trial sources (agripro_trials, gh_plot_reports) so the monthly cron refreshes them alongside the variety sources. gh_plot_reports is the heaviest single source (~4,600 docs @ 1 req/sec ≈ 70 min); runs late so an earlier failure doesn't waste time before failing. Commit-message variable count expanded to surface the trial counts. docs_mcp/lessons.md — new "trial-data" section telling the agent: - The two surfaces (search_docs = identity, search_trials = perf) are complementary; how to route a farmer question to each. - What's indexed (GH plot reports cross-vendor, AgriPro regional PDFs) vs what's not (Bayer per-variety trials, NK yield results, Pioneer, university extension trials). - Recommended workflow: search_trials → identify top performers → lookup_variety on each to verify identity → don't fabricate. - How to read a GH plot report (per-column headers vary by crop: corn/soy use Yield/MST/Test Weight, silage uses Ton/Acre + Milk + Beef columns). - Single-data-point caveat: one plot is one cooperator's field; look across multiple plots for a robust recommendation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -83,10 +83,21 @@ jobs:
|
||||
if: ${{ inputs.sources == '' || contains(inputs.sources, 'agripro') }}
|
||||
run: python -m scrape.runner --source agripro --force
|
||||
|
||||
- name: Scrape AgriPro regional trial PDFs
|
||||
if: ${{ inputs.sources == '' || contains(inputs.sources, 'agripro_trials') }}
|
||||
run: python -m scrape.runner --source agripro_trials --force
|
||||
|
||||
- name: Scrape Golden Harvest plot reports (cross-vendor yield trials)
|
||||
if: ${{ inputs.sources == '' || contains(inputs.sources, 'gh_plot_reports') }}
|
||||
# Heaviest single source — ~4,600 docs at 1 req/sec ≈ 70 min.
|
||||
# Wraps the bulk of CI time; runs late so an earlier failure
|
||||
# doesn't waste 70 min before failing.
|
||||
run: python -m scrape.runner --source gh_plot_reports --force
|
||||
|
||||
- name: Scrape Beck's PFR research corpus
|
||||
if: ${{ inputs.sources == '' || contains(inputs.sources, 'becks_pfr') }}
|
||||
# Heaviest source — ~2,089 docs via public Sanity GROQ.
|
||||
# No auth, but rate-limit ourselves to be polite.
|
||||
# Deferred (returns 0 cleanly from a stub) — implementation
|
||||
# pending. Public Sanity GROQ at mc8v24rf.api.sanity.io.
|
||||
run: python -m scrape.runner --source becks_pfr --force
|
||||
|
||||
# ---- Commit corpus changes + retry-on-race -----------------
|
||||
@@ -107,8 +118,10 @@ jobs:
|
||||
n_gh=$(find corpus/golden_harvest -name '*.json' 2>/dev/null | wc -l)
|
||||
n_nk=$(find corpus/nk -name '*.json' 2>/dev/null | wc -l)
|
||||
n_ag=$(find corpus/agripro -name '*.json' 2>/dev/null | wc -l)
|
||||
n_agt=$(find corpus/agripro_trials -name '*.json' 2>/dev/null | wc -l)
|
||||
n_ghpr=$(find corpus/gh_plot_reports -name '*.json' 2>/dev/null | wc -l)
|
||||
n_pfr=$(find corpus/becks_pfr -name '*.json' 2>/dev/null | wc -l)
|
||||
git commit -m "monthly refresh: ${ts} — bayer=${n_bayer} gh=${n_gh} nk=${n_nk} agripro=${n_ag} pfr=${n_pfr}"
|
||||
git commit -m "monthly refresh: ${ts} — bayer=${n_bayer} gh=${n_gh} nk=${n_nk} agripro=${n_ag} ag_trials=${n_agt} gh_plot_reports=${n_ghpr} pfr=${n_pfr}"
|
||||
attempt=1
|
||||
while [ $attempt -le 3 ]; do
|
||||
if git push; then
|
||||
|
||||
Reference in New Issue
Block a user