The monthly refresh has failed every month since 2026-06-01, always at
exactly 3:00:00, on two runner hosts and three act_runner versions. The
job container is created as `/bin/sleep 10800` (act_runner's default
`runner.timeout`), so at 3 h it vanishes mid-step and the run dies with
the misleading `container "GITEA-ACTIONS-TASK-..." does not exist` /
`docker daemon ping ... context deadline exceeded`.
It was never going to fit. `--force` re-fetched all 11,415 candidate
registrations every month. Profiling the 2026-09-01 log (2,198 products
in 2 h 44 m before the kill):
- ~7,300 discarded as not-row-crop at a 1.31 s median, essentially all
of it the hard-coded REQUEST_DELAY_SECONDS=1.1 sleep — ~2.8 h/month
spent deciding to throw things away.
- ~4,070 written at a 6.6 s median but a 15.0 s MEAN: individual PDFs
burned minutes (100-1262 = 689 s, 241-441 = 602 s).
- Extrapolated full run: 15-20 h, 5-7x the cap.
Three changes, measured end-to-end against a throwaway corpus:
1. Bound PDF downloads. httpx timeouts are per-read, so a body trickling
in at ~50 KB/s never trips them and download_pdf() could hang as long
as EPA kept dribbling. It now streams with an overall 180 s deadline
and 2 attempts instead of 4. Verified live: the 33.8 MB 241-441 label
tripped the deadline at 8.9 MB on attempt 1 and completed on the
retry, where before it cost 600 s.
2. Make it incremental. A committed filter cache remembers not-row-crop
verdicts (TTL 180 d, with a deterministic +/-30 d per-product jitter
so a cold run's verdicts do not all expire in the same month and
resurrect this bug). Products already on disk are re-downloaded only
when EPA reports a new label acceptance date or PDF URL, so dropping
--force costs no freshness — unlike the old skip-if-exists check,
which never noticed a revision.
3. Parallelise. --workers (default 6) with a shared 5 req/sec ceiling,
replacing the per-request sleeps: faster without being ruder.
Measured on 300 products: cold 4.6 min -> warm 1.0 min, 0 errors
(0 wrote / 80 unchanged / 220 filtered-cached). A steady-state monthly
refresh should land at ~15-20 min against a 170-minute job timeout,
which now fails legibly instead of letting the container disappear.
The workflow commits scrape/state so the next run starts warm, but
decides `changed` from the corpus paths alone — otherwise the cache
would fake a corpus diff every month and trigger a needless reindex
and image push.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FnVuG79cYPcRLTp4pC8ujR
Run #122 finished green-on-everything-that-matters: indexing,
docker login (REGISTRY_TOKEN fix worked), build + push, and the
package-link API call all succeeded. The image is published with
all four expected tags: latest, c5ed5560fc, corpus-2026.05.24,
a97107de46 (manual earlier push).
Only the final GC step failed with HTTP 403 enumerating
/packages/.../versions — the PAT we use as REGISTRY_TOKEN has
push/pull scope but not the broader package-admin scope needed
to list + delete old versions.
GC is housekeeping, not part of the publish path. Marking it
continue-on-error: true keeps the whole run green so monitoring
can rely on "red = real problem." Both workflows get the same
treatment.
Followup TODO baked into the workflow comments: mint a separate
PAT with admin:package scope and add it as a second secret
(PACKAGES_ADMIN_TOKEN) — then point the GC step at it. Then
remove continue-on-error.
Workflow-only commit, doesn't trigger image-only.yml (path filter
excludes .gitea/**).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two rapid commits today (2acba0a → c5ed556) queued back-to-back
runs of the same workflow. Each run is ~90 min (the reindex is the
long pole), and the older commit's image just gets overwritten by
the newer one anyway — so the older run is pure waste.
concurrency:
group: image-only
cancel-in-progress: true
cancels any in-flight run on the same workflow when a new push
lands. Future-me will thank present-me.
(The workflow-file change doesn't itself trigger image-only.yml
since .gitea/workflows/** isn't in the paths: filter — so this
commit doesn't kick off a third run.)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cloudflare in front of git.jpaul.io caps HTTP request bodies at
100 MB, which kills container blob pushes for our 6 GB image
(Chroma layer alone is ~2 GB). Per the template gotcha doc:
Push via LAN endpoint (192.168.0.2:1234, plain HTTP, in the
Gitea host's insecure-registries list).
Pull via public hostname (git.jpaul.io) — pull response bodies
aren't capped.
REGISTRY_PUSH: 192.168.0.2:1234
REGISTRY_PULL: git.jpaul.io (unchanged; used for the package-link API)
This matches how hvm-docs / morpheus-docs / opsramp-docs / zerto-docs
CI workflows push successfully on the same Gitea host.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Dockerfile: self-contained image with corpus + Chroma + BM25 baked
in. Drawbar's compose pulls + runs without volume mounts. Built from
sources.json (labels schema), PRODUCT_NAME=crop_chem by default,
HYBRID_SEARCH=true (always-on for production quality). RERANK_URL +
OLLAMA_URL get set at compose time.
.gitea/workflows/refresh.yml: monthly cron (1st @ 06:00 UTC) does
full scrape → reindex → image push. Scrapes Bayer (~30 min) +
EPA PPLS row-crop filtered (~7h). Skips reindex+push if no corpus
diff. Tags pushed: :latest, :<sha12>, :corpus-<YYYY.MM.DD>.
.gitea/workflows/image-only.yml: on-demand or auto on code-only
pushes to main (paths: docs_mcp/, rag/, scrape/, requirements.txt,
Dockerfile, sources.json). Reindexes from committed corpus, builds
image, pushes. ~10 min vs ~9h full refresh.
.gitignore: corpus/ now COMMITTED (4,159 labels, 265 MB of .md +
sidecars). Lets image-only.yml rebuild indexes without re-scraping.
chroma/ + bm25/ still gitignored (regenerable binary indexes).
.dockerignore: drops venv, eval results, PLAN/README/CLAUDE.md,
deploy/, .git/ — keeps the image lean. corpus + chroma + bm25
explicitly NOT in dockerignore (those go INTO the image).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
First dispatch on the empty template failed at Chroma collection
creation because PRODUCT_NAME was the literal string "<product>"
(YAML doesn't expand placeholders), and Chroma rejects collection
names containing characters outside [a-zA-Z0-9._-]:
chromadb.errors.InvalidArgumentError: Validation error: name:
Expected a name containing 3-512 characters from [a-zA-Z0-9._-],
starting and ending with a character in [a-zA-Z0-9]. Got:
<product>_docs
Same fix as the IMAGE env: derive from the repo name dynamically
via ${{ github.event.repository.name }}. Cloners can still override
explicitly, but a fresh clone now runs the index-rebuild step
cleanly out of the box.
Verified by re-dispatch — should fail next at docker login (placeholder
REGISTRY_PUSH hostname), which is the next-expected fail point and a
real per-deployment config the cloner has to fill in.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Both workflows had a static IMAGE env (<owner>/<product>-docs-mcp)
and a static --package arg in the GC step. Switch both to Gitea
Actions context variables so a clone of the template into any repo
name works on the first CI run without find/replace:
IMAGE: ${{ github.repository_owner }}/${{ github.event.repository.name }}
--owner ${{ github.repository_owner }}
--package ${{ github.event.repository.name }}
Also add the "Link container package to this repo" step that was
missing from the template (and which, naively copy-pasted from the
reference build, would have linked everything back to docs-mcp-
template). The new step derives owner + package + link-target all
from the running repo's context.
The github.* namespace is Gitea Actions' inherited GitHub-Actions
context — values come from the Gitea server, not github.com. Same
mechanism the reference build's $GITHUB_SHA tag-builder uses.
CLAUDE.md updated to note that image and package naming are
repo-derived; only registry endpoints and the Ollama URL need
per-clone editing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
claudeandClaude Opus 5
justinandClaude Opus 4.7