crop-chem-docs/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Purpose

This is a template for building an MCP server over a product's public documentation. When you (Claude) are working in a clone of this repo, you are helping the user implement one specific product's docs MCP — not editing the template itself.

Read PLAN.md first. It's the canonical build guide and lays out 13 phases. Most user requests will be "implement Phase N" or "we hit a bug in Phase N." Identify the phase before doing anything else.

Working with this template

Identifying the current phase

When the user clones this template and starts working, figure out which phase they're on by inspecting:

Signal	Likely phase
corpus/ doesn't exist	Phase 1 (scraper) — they need to build it before anything else works
corpus/ exists, chroma/ doesn't	Phase 2 (indexing)
Indexes exist, only search_docs / get_page / list_versions implemented	Phase 3 (server skeleton done; next: Dockerfile + CI)
No Dockerfile or .gitea/workflows/ updated	Phase 4–5
RERANK_URL env unset in compose	Phase 6 not done
HYBRID_SEARCH env unset, no rag/bm25.py content	Phase 8 not done
No eval/results/ directory	Phase 7 not done
find_doc_inconsistencies is a commented-out stub in docs_mcp/server.py	Phase 12
No corpus/.digest/ produced by CI	Phase 13

When in doubt, ask the user: "Which phase from PLAN.md are we working on?"

The scaffolded server has stubs

docs_mcp/server.py ships with three working tools (search_docs, get_page, list_versions) and signature-only stubs for the phase-specific tools. The stubs raise NotImplementedError with a phase hint in the docstring. When implementing a phase, you'll be filling these bodies in — DO NOT change the signatures unless the user has a specific reason. Signatures are the public contract between the MCP and its clients (Claude Desktop, Claude Code, Cursor, etc.).

Layout

.
├── PLAN.md                       # Read first. Phase-by-phase build guide.
├── README.md                     # Quick-start summary.
├── CLAUDE.md                     # This file.
├── requirements.txt
├── Dockerfile
├── deploy/docker-compose.yml
├── .gitea/workflows/
│   ├── refresh.yml               # Weekly cron: scrape + index + image
│   └── image-only.yml            # On-demand: code-only ship cycle
├── scrape/                       # Phase 1 — product-specific scraper here
│   └── changelog.py              # Reusable: --json, --history-out
├── rag/                          # Phase 2/8 — indexing
│   ├── embeddings.py             # Ollama embedder (swappable)
│   ├── chunk.py                  # Page → chunks (adjust per page format)
│   ├── index.py                  # Builds Chroma + BM25
│   └── bm25.py                   # SQLite FTS5 lexical index
├── docs_mcp/                     # Phase 3+ — MCP server
│   ├── server.py                 # MCPServer + tool definitions
│   └── usage.py                  # TimedCall telemetry
├── eval/                         # Phase 7 — golden-query harness
│   ├── queries.jsonl.example
│   ├── retrievers.py
│   └── run_eval.py
├── scripts/                      # Standalone ops scripts
│   ├── usage_report.py
│   └── registry_gc.py
└── deploy/
    └── docker-compose.yml

Conventions

Tool docstrings are user interface

The text in @mcp.tool() docstrings is what the LLM sees and uses to decide whether to call the tool. Treat it like a button label. "Use when...", "Call proactively whenever..." phrasings work well. Don't bury the headline in implementation notes.

Defensive fallback for retrieval components

The reranker, BM25 index, and any external dependency must fail gracefully:

• Catch the specific exception type
• Log a warning with enough info to debug
• Fall back to a working baseline (dense-only, no reranker, etc.)
• Never block a search_docs call on a single failure

The user's MCP is in front of real people; partial degradation beats a 500.

Verify retrieval changes with the eval harness

Any change that touches retrieval (new embedder, chunker tweak, reranker model, filter shape) ships with eval numbers in the commit message. Don't ship retrieval changes on vibes. If eval/queries.jsonl isn't populated yet, populate it before changing retrieval — it's the most important file in the repo.

Standard infrastructure choices

These are reasoned defaults — only deviate if you have a specific need:

• Embedding model: nomic-embed-text via Ollama (768-dim, free, on-prem)
• Reranker: jina-reranker-v2-base GGUF via llama.cpp /v1/rerank endpoint
• Vector store: Chroma PersistentClient
• Lexical store: SQLite FTS5 (stdlib)
• Fusion: Reciprocal Rank Fusion with k=60
• Transport: streamable-HTTP in prod, stdio for local dev
• MCP framework: mcp.server.mcpserver.MCPServer (mcp 2.x), run with stateless_http=True
• Container deploy: Watchtower auto-pull on :latest, rollback via :<sha12> pin

Naming the product

The template uses PRODUCT_NAME env var (defaults to "myproduct") throughout. Set it on first build. References show up in:

• docs_mcp/server.py — MCPServer(f"{PRODUCT_NAME}-docs")
• Collection name (<product>_docs)
• BM25 db filename
• Tool names that include the product name (e.g., the _api_lessons tool — convention is to name it <product>_api_lessons)

Use lowercase, underscores not hyphens, since it ends up in tool identifiers that the LLM reads.

Image name and package linking are repo-name-derived

You do NOT need to edit the IMAGE env or the --package arg in the workflows. Both derive from the repo at runtime via ${{ github.repository }} and ${GITHUB_REPOSITORY##*/}. Do not use ${{ github.event.repository.name }} — it resolves to EMPTY on schedule events (Gitea >= 1.27), which silently builds the tag <registry>/<owner>/:latest and fails the build after the full scrape; the Verify image name resolves step guards it. So a clone into a repo named my-product-docs automatically pushes the container as <owner>/my-product-docs:latest and links the package to its own repo. (github.* is Gitea Actions' inherited GitHub-Actions namespace — the values come from the Gitea server, no github.com involvement.)

The only workflow placeholders you still have to replace per clone are the registry endpoints (REGISTRY_PUSH, REGISTRY_PULL) and the Ollama URL, because those depend on the deployment environment, not the repo identity.

Common commands

# Set up dev environment
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Run the MCP server locally for Claude Desktop dev
python -m docs_mcp.server --transport stdio

# Run as HTTP for integration testing
python -m docs_mcp.server --transport streamable-http --port 8000

# Rebuild Chroma + BM25 indexes from corpus
python -m rag.index --rebuild

# Rebuild only BM25 (fast iteration)
python -m rag.index --bm25-only

# Run the eval harness
python -m eval.run_eval --queries eval/queries.jsonl --output eval/results/baseline.md

# Generate changelog summary (called by CI, useful locally too)
python -m scrape.changelog --cached
python -m scrape.changelog --history-out corpus/.digest/history.jsonl --history-days 120

Gotchas (carried forward from the reference build)

• fetch-depth: 0 on actions/checkout@v4 in both workflows. Default is shallow; history-walking steps (changelog, digest) silently produce empty output otherwise. This is the #1 thing people miss.
• Reranker per-pair token limit: jina-reranker GGUF rejects the ENTIRE batch if any doc exceeds n_ctx_train=1024. Truncate docs to ~2000 chars before sending to rerank. Full chunk text still goes back to the user; truncation is reranking-only.
• stateless_http=True: critical for production hosting behind Watchtower auto-updates. Without it, every container recreate produces a 404 storm from clients with stale session IDs. Under mcp 2.x this is a mcp.run() kwarg, not a constructor arg — see main() in docs_mcp/server.py.
• mcp 2.x, not 1.x: mcp.server.fastmcp was removed in 2.0.0. This server uses mcp.server.mcpserver.MCPServer; mcp.settings no longer exists (host/port/transport_security are run() kwargs). PLAN.md still describes the 1.x API — it tracks the upstream template, not this repo.
• Runner shell is /bin/sh (dash): no ${VAR::N} substring expansion in workflow scripts. Use cut/awk/printf.
• Cloudflare 100 MB body cap: if pushing through a Cloudflare- fronted registry, push via LAN endpoint, pull via public hostname. Same registry, different URLs.

When the user says...

User says	You do
"Let's start building" / "set up the project"	Read PLAN.md Phase 0; create dirs, requirements.txt, etc. Confirm Python version and existing tooling.
"Build the scraper" / "scrape the docs"	Read PLAN.md Phase 1. Find the upstream portal's underlying API by sniffing; AVOID headless-browser solutions unless the API path is truly closed.
"Get retrieval working" / "make search work"	Read PLAN.md Phase 2-3. Implement chunking, embedder, Chroma indexer, then the three baseline tools.
"Add a reranker"	Read PLAN.md Phase 6. Stand up the llama.cpp sidecar, implement _rerank(). Verify with the eval harness.
"Search is missing X queries"	Run the eval harness first to confirm the failure. Then consider: rich chunk-0 rewrites, hybrid retrieval, curated knowledge layer. Don't just tune cosine.
"Let's add hybrid search"	Read PLAN.md Phase 8. Only after you've established the failure mode with eval queries — hybrid is not free.
"I want a 'what changed' tool"	Read PLAN.md Phase 13. Don't try to do this at runtime — pre-bake the history JSONL at CI time.

Out-of-scope concerns (don't try to solve here)

• Reverse proxy / TLS termination — outside the repo. User picks Caddy / Cloudflare Tunnel / nginx / Traefik based on their infra.
• MetaMCP or other gateway — outside the repo. Optional, only matters when running multiple MCPs.
• GPU container orchestration — outside the repo. Pattern is one Ollama container per GPU; the indexer load-balances. Document it in deploy/ but don't build it in this template.
• Email/blog delivery for weekly_digest — out of scope per PLAN.md ("Out of scope" section). Add a separate script in scripts/ if/when the user asks.