The clean refresh (#15483) aborted the whole scrape on a transient upstream
blip:
requests.exceptions.ConnectionError:
('Connection aborted.', RemoteDisconnected('Remote end closed ...'))
_get() retried HTTP error *statuses* (429/500/502/503/504) but not
connection-level *exceptions*, so a single dropped connection — which the
HPE portal does intermittently under the --force re-scrape of 2300+ pages
at concurrency 6 — killed the entire weekly run. Prior runs just got lucky.
- _get(): wrap the request in try/except requests.exceptions.RequestException
and retry with the existing exponential backoff (covers RemoteDisconnected,
timeouts, chunked-encoding errors).
- scrape_toc_bundle(): a page that still fails after all retries is logged
and skipped (its previously-committed .md is kept) instead of aborting the
bundle. A broad outage still surfaces as a high failed-count in the log.
Verified: unit test confirms _get retries a RemoteDisconnected then succeeds;
happy-path scrape unaffected.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
claudeandClaude Opus 4.8