justin
60657aa6df
The farmer-advisor consumer only cares about US row crops, so the EPA
scraper now drops products without at least one row-crop site in the
PPLS API response. Filter is on by default; --no-row-crop-filter
overrides for one-off broader pulls.
Filter shape:
- Word-boundary regex match against each entry in the API's `sites`
array (e.g., "SOYBEANS (FOLIAR TREATMENT)" → keep, "SHIPS, BOATS,
SHIPHOLDS" → drop even though it contains "OATS" as substring).
- Allowlist covers the major US row + small-grain + oilseed + sugar/
fiber crops, plus alfalfa as a common rotation crop. See
ROW_CROP_KEYWORDS in scrape/sources/epa_ppls.py for the full list.
Cost model:
- 102K PPIS rows still need one API call each (no bulk filter
available upstream), so enumeration still takes ~28h at 1 req/sec.
- But PDF downloads drop from ~67K → ~5-10K (estimated row-crop
hit rate), saving ~17h wall time and ~60GB disk on a full backfill.
Smoke test (4 mixed reg nos):
524-475 Roundup Ultra → kept (CORN/SOYBEANS/COTTON sites)
524-591 Warrant → kept (CORN/SOYBEANS/SORGHUM sites)
100-1486 Advion Cockroach → filtered (building/transport sites only)
432-1276 (Bayer pet flea) → filtered (no row crops)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
22 lines
1.0 KiB
JSON
22 lines
1.0 KiB
JSON
[
|
|
{
|
|
"id": "bayer",
|
|
"title": "Bayer Crop Science US — Product Labels",
|
|
"type": "manufacturer",
|
|
"homepage": "https://www.cropscience.bayer.us",
|
|
"scraper": "scrape.sources.bayer",
|
|
"scraper_version": "0.1.0",
|
|
"license_note": "robots.txt explicitly permits scraping for AI retrieval-augmented generation (verified 2026-05)"
|
|
},
|
|
{
|
|
"id": "epa_ppls",
|
|
"title": "EPA Pesticide Product Label System",
|
|
"type": "regulator",
|
|
"homepage": "https://ordspub.epa.gov/ords/pesticides/f?p=PPLS:1",
|
|
"scraper": "scrape.sources.epa_ppls",
|
|
"scraper_version": "0.1.0",
|
|
"license_note": "US federal government — public domain (no ToS restriction)",
|
|
"scope_filter": "row-crop only — products with at least one site matching CORN, MAIZE, POPCORN, SOYBEAN(S), COTTON, WHEAT, RICE, SORGHUM, MILO, BARLEY, OATS, RYE, SUNFLOWER(S), PEANUT(S), SUGAR BEET(S), DRY/FIELD BEAN(S), CANOLA, RAPESEED, or ALFALFA (word-boundary match). Pass --no-row-crop-filter to scrape the full PPLS universe."
|
|
}
|
|
]
|