claudeandClaude Opus 4.8
1d3e52a5ed
Some HPE pages carry information only inside an image. The Morpheus release schedule / support-lifecycle matrix (sf000111242en_us) is the case that surfaced this: the entire "which version ships when, when a stream hits Maintenance/EOL" table is a JPEG, so html_to_md dropped every value and retrieval could never answer "when does 8.x.x reach end of life". New scrape/vision.py (opt-in via VISION_OCR) transcribes qualifying images with a local Ollama vision model and appends a labeled markdown block to the page so the text is chunked, embedded and retrieved. Model/prompt chosen against ground truth on the matrix (14x4 cells): - qwen2.5vl:7b + a bare "transcribe the table" prompt: 55-56/56, ~18s, GPU-resident, and prompt-ROBUST (accurate without hand-tuning). - gemma3:12b needs an exact "read row by row, keep cells aligned" prompt or it shifts a column by one row; qwen2.5vl:32b is no more accurate and 5x slower (CPU spill). Both documented in the code. - Note: self-consistency (2 samples must agree) only catches RANDOM flakiness — a wrong read is stable across seeds — so every block also ships a "verify against the source image" caveat. Reliability/cost: - content-hash cache in corpus/.vision-cache/ (committed) — each unique image OCR'd once ever, shared across version bundles. - VISION_MAX_NEW bounds NEW OCRs per run so the first pass can't balloon into hours; the cache fills incrementally. Deferred count is logged. - every failure path degrades to the pre-vision behavior; never blocks the scrape. Also: - add the morpheus_release_schedule bundle (sf single-doc) + 2 eval golden queries for it. - fetch_single_doc: title falls back to the bundle title (not docId) when a page has no <h1> (sf solution articles). - Pillow in requirements-vision.txt (scrape-only; kept out of the server image), installed + VISION_* wired into refresh.yml. Verified locally: matrix transcribes to the exact 14-row table; second run is a cache hit (ocr=0). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LFowQzJu7k97QLCRDSAeh1
docs-mcp-template
A reusable template for building hosted MCP servers over a product's public documentation. Distilled from one production build; everything product-specific has been factored out.
The end product is a streamable-HTTP MCP server with ~15 tools that any LLM client (Claude Desktop, Claude Code, Cursor, Copilot) can call to answer questions against the docs, surface what changed recently, find inconsistencies, and (optionally) submit doc bugs back upstream.
What's here
- PLAN.md — comprehensive build guide. Phased approach (13 phases, ~2–3 weeks of focused work for the full stack). Includes the design decisions, the gotchas, and a per-product customization checklist.
- Scaffolded skeleton — working FastMCP server with stub tools,
Dockerfile, docker-compose, CI workflows, eval harness layout,
usage logging. Everything you need to
git cloneand start filling in the product-specific bits.
Quick start
git clone https://git.jpaul.io/justin/docs-mcp-template.git my-product-docs
cd my-product-docs
git remote remove origin # detach from template
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# Read PLAN.md before doing anything else. Pay particular attention to
# Phase 1 (scraper) — that's the most product-specific phase.
# Run the stub server (no corpus yet — just verifies the wiring):
python -m docs_mcp.server --transport stdio
Repo layout
.
├── PLAN.md # The build guide. Read first.
├── README.md
├── requirements.txt
├── Dockerfile
├── .gitignore
├── .gitea/workflows/
│ ├── refresh.yml # Weekly scrape + index + image push
│ └── image-only.yml # On-demand code-only ship
├── scrape/
│ ├── README.md # Product-specific scraper goes here
│ └── changelog.py # Reusable: --json, --history-out
├── rag/
│ ├── embeddings.py # Ollama embedder, swappable
│ ├── chunk.py # Chunker — adjust per page format
│ ├── index.py # Builds Chroma + (optionally) BM25
│ └── bm25.py # SQLite FTS5 lexical index
├── docs_mcp/
│ ├── server.py # FastMCP server with stub tools
│ └── usage.py # TimedCall + JSONL telemetry
├── eval/
│ ├── queries.jsonl.example # Curate ~25 hand-labeled queries
│ ├── retrievers.py # Retriever protocol + implementations
│ └── run_eval.py # MRR / Recall@k / nDCG@k harness
├── scripts/
│ ├── usage_report.py # Standalone log analyzer
│ └── registry_gc.py # Container registry cleanup
└── deploy/
└── docker-compose.yml # Hosting stack: MCP + reranker + Watchtower
What's product-specific (must implement)
scrape/— the scraper itself. The template gives you the corpus layout contract and a workingchangelog.py; the actual extraction logic is yours.- The corpus on disk (gitignored; rebuilt by CI).
- The reranker GGUF model and llama.cpp container (commented in
deploy/docker-compose.yml). - The reverse proxy / TLS layer in front of the public endpoint.
- The hand-curated knowledge surface (your product's API gotchas, example scripts, anything the LLM should know that the docs don't say).
What's NOT product-specific (works as-is)
- FastMCP server skeleton + tool decoration pattern
- Chroma + Ollama embedding pipeline
- BM25 / SQLite FTS5 lexical index
- Hybrid retrieval (RRF) + reranker integration
- Eval harness (Retriever protocol, MRR/Recall/nDCG)
- Usage logging (TimedCall, JSONL, daily rotation)
- CI workflow shape (weekly + on-demand, retry-on-race, three-tag image scheme)
- Registry GC script
- Standard tools:
search_docs,get_page,list_versions,diff_versions,bundle_changelog,weekly_digest,find_doc_inconsistencies,submit_doc_bug, etc.
License
Internal template. Adjust before publishing.