Two new trial sources: LG Seeds + AgriGold plot reports (+2,307 cross-vendor yield trials)

Adds the **first non-Syngenta trial coverage** to the corpus:

| Source | Docs | Publisher | URL pattern |
|---|---|---|---|
| lg_plot_reports | 1,304 | LG Seeds (AgReliant) | lgseeds.com/performance/{crop} JSON XHR |
| agrigold_plot_reports | 1,003 | AgriGold (AgReliant) | agrigold.com/{crop}/performance/{crop}-yield-results |

Total trial coverage now: gh_plot_reports (4,299) + agripro_trials (14) +
lg_plot_reports (1,304) + agrigold_plot_reports (1,003) = 6,620 trial docs.

**Both scrapers follow the gh_plot_reports template** — same RateLimitedSession
primitive, same TrialResult/PlotReport dataclass shape, same data_type="trial"
sidecar convention. The trial chunker (`rag/chunk.py:_render_trial_chunk`) is
extended to recognize both new sources; they share `_render_gh_plot_chunk`
since their sidecars are structurally identical (just different brand label).

**LG specifics:**
- POST `/performance/{crop}/GetPlots/` returns sparse listing (id, year, lat/lng)
- GET `/performance/{crop}/GetPlotData/?PlotId=X&IsSilage=Y` returns full detail
  with state, cooperator, planting/harvest dates, and **top-5 hybrids** (LG +
  competitors). Top-5 is what LG publishes publicly; not the full ranking.
- 4 crops: corn (963), soybeans (287), sorghum (10), silage (50) — first
  alfalfa absent because LG doesn't run alfalfa plots; that's variety-only data.
- 301 gotcha: www.lgseeds.com redirects to lgseeds.com which drops POST body,
  so the scraper hits the apex host directly.

**AgriGold specifics:**
- Listing: GET `/{crop}/performance/{crop}-yield-results?harvestYear={year}`
  (server-rendered HTML, ~1MB; 408 corn plots in 2025 alone)
- Detail: GET `/{crop_url}/performance/{slug}/{plot_id}` returns the **full
  ranking** (not just top-5) plus rich plot management metadata: tillage,
  previous crop, fungicide, herbicide, insecticide, irrigation, soil type,
  row width, population. Most metadata-rich of the three trial sources.
- Soybean URL slug is singular: `/soybeans/performance/soybean-yield-results/`
- Columns: Rank | Brand | Product | Trait | Ck | H20 (moisture) | Test Wt. |
  Yield | Adj Yield (check-adjusted)
- 2 crops: corn (849) + soybeans (157)

**Indexer needs no changes** — `rag/index.py` auto-discovers any directory
under corpus/ and routes by data_type. Both new sources flow into the
existing trial collection and surface via `search_trials`.

Years scraped: 2024+2025 (matching gh_plot_reports baseline). 2023 is
available via `--include-2023` on either scraper for future backfill.
This commit is contained in:
2026-05-26 22:26:24 -04:00
parent db1567f84a
commit b98965a68a
4630 changed files with 449945 additions and 4 deletions
+34
View File
@@ -143,6 +143,40 @@
"tos_check_date": "2026-05-25",
"schema_notes": "PDF tables of varieties tested per region per year. pdfplumber for table extraction.",
"data_type": "trial"
},
{
"name": "lg_plot_reports",
"vendor": "AgReliant Genetics",
"brand_aggregator": "LG Seeds publishes",
"crops": [
"corn",
"soybeans",
"sorghum",
"silage"
],
"verdict": "green",
"expected_count": 1310,
"base_url": "https://lgseeds.com",
"scope_filter": "Cross-vendor plot-report data exposed by lgseeds.com/performance/{crop} JSON XHR (GetPlots + GetPlotData). 2024+2025 baseline (older 2023 deferred to --include-2023). Top-5 hybrids per plot (LG + competitors), not full ranking — that's what LG publishes publicly.",
"tos_check_date": "2026-05-26",
"schema_notes": "JSON API behind www→apex redirect; POST GetPlots returns list with year+coords, GET GetPlotData returns top-5 + state/cooperator/dates. 301 redirect drops POST body so hit `lgseeds.com` (no www). Top-5 means each plot is partial coverage — multiple plots cover the same site/cooperator with different LG hybrid lineups.",
"data_type": "trial"
},
{
"name": "agrigold_plot_reports",
"vendor": "AgReliant Genetics",
"brand_aggregator": "AgriGold publishes",
"crops": [
"corn",
"soybeans"
],
"verdict": "green",
"expected_count": 1006,
"base_url": "https://www.agrigold.com",
"scope_filter": "Cross-vendor plot reports at agrigold.com/{crop}/performance/{crop}-yield-results, year-filtered via ?harvestYear=. Detail page exposes FULL ranking (not just top-5) plus rich plot management (tillage, prev crop, fungicide, soil type, irrigation). 2024+2025 baseline.",
"tos_check_date": "2026-05-26",
"schema_notes": "Server-rendered HTML; detail URL is /{crop_url}/performance/{slug}/{id}. Soybeans URL slug is singular: /soybeans/performance/soybean-yield-results/{id}. Columns: Rank, Brand, Product, Trait, Ck, H20 (moisture %), Test Wt., Yield, Adj Yield. Most metadata-rich of the three trial sources.",
"data_type": "trial"
}
],
"_excluded_sources": [