fix(scrape): retry connection drops; tolerate single-page failures #8

2026-08-06T11:55:03-04:00

claude commented

2026-08-06 11:55:03 -04:00

Problem

The clean refresh (#15483) aborted the whole scrape on a transient upstream network blip:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected(...))
  at scrape/runner.py:_get → fetch_toc_page

_get() retried HTTP error statuses (429/500/502/503/504) but not connection-level exceptions — so one dropped connection (which the HPE portal does intermittently under the --force re-scrape of 2,300+ pages at concurrency 6) killed the entire weekly run. Not caused by the vision/allowlist work — latent fragility that prior runs got lucky on.

Fix

_get(): wrap the request in try/except requests.exceptions.RequestException and retry with the existing exponential backoff (covers RemoteDisconnected, timeouts, chunked-encoding errors).
scrape_toc_bundle(): a page that still fails after all retries is logged + skipped (its committed .md is kept) instead of aborting the bundle. A broad outage still surfaces as a high failed-count.

Verified

Unit test confirms _get retries a RemoteDisconnected then succeeds; happy-path scrape unaffected.

## Problem The clean refresh (#15483) **aborted the whole scrape** on a transient upstream network blip: ``` requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected(...)) at scrape/runner.py:_get → fetch_toc_page ``` `_get()` retried HTTP error *statuses* (429/500/502/503/504) but **not connection-level exceptions** — so one dropped connection (which the HPE portal does intermittently under the `--force` re-scrape of 2,300+ pages at concurrency 6) killed the entire weekly run. Not caused by the vision/allowlist work — latent fragility that prior runs got lucky on. ## Fix - **`_get()`**: wrap the request in `try/except requests.exceptions.RequestException` and retry with the existing exponential backoff (covers `RemoteDisconnected`, timeouts, chunked-encoding errors). - **`scrape_toc_bundle()`**: a page that still fails after all retries is logged + skipped (its committed `.md` is kept) instead of aborting the bundle. A broad outage still surfaces as a high failed-count. ## Verified Unit test confirms `_get` retries a `RemoteDisconnected` then succeeds; happy-path scrape unaffected.

claude added 1 commit 2026-08-06 11:55:03 -04:00

fix(scrape): retry connection drops; tolerate single-page failures b129ff1ef9

The clean refresh (#15483) aborted the whole scrape on a transient upstream
blip:
  requests.exceptions.ConnectionError:
    ('Connection aborted.', RemoteDisconnected('Remote end closed ...'))

_get() retried HTTP error *statuses* (429/500/502/503/504) but not
connection-level *exceptions*, so a single dropped connection — which the
HPE portal does intermittently under the --force re-scrape of 2300+ pages
at concurrency 6 — killed the entire weekly run. Prior runs just got lucky.

- _get(): wrap the request in try/except requests.exceptions.RequestException
  and retry with the existing exponential backoff (covers RemoteDisconnected,
  timeouts, chunked-encoding errors).
- scrape_toc_bundle(): a page that still fails after all retries is logged
  and skipped (its previously-committed .md is kept) instead of aborting the
  bundle. A broad outage still surfaces as a high failed-count in the log.

Verified: unit test confirms _get retries a RemoteDisconnected then succeeds;
happy-path scrape unaffected.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1

claude merged commit 8a67366ad6 into main

2026-08-06 11:55:10 -04:00

claude deleted branch fix/scrape-retry-connection-errors

2026-08-06 11:55:10 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: justin/morpheus-docs#8