initial: docs-mcp-template — build guide + scaffolded server

Template for building hosted MCP servers over a product's public
documentation. Distilled from one production build; everything
product-specific has been factored out.

Contents:

- PLAN.md — comprehensive build guide. 13 phases from project
  skeleton through weekly_digest. Includes the gotchas
  ("fetch-depth: 0 always", reranker per-pair token limit,
  Cloudflare body cap, dash-not-bash on Gitea runners), the
  decisions worth carrying forward, and a per-product
  customization checklist.

- CLAUDE.md — guidance for Claude Code working in a clone of this
  template. Phase identification table, conventions (env-gating +
  operator confirmation for side-effecting tools, defensive
  fallback for retrieval components), common commands.

- README.md — quick-start summary.

Scaffolded code (all signature-stable, with NotImplementedError
stubs where phase-specific work is required):

  docs_mcp/server.py    FastMCP server, stateless_http=True, with
                        search_docs / get_page / list_versions
                        baseline tools and commented stubs for the
                        rest of the phase set.
  docs_mcp/usage.py     TimedCall telemetry, JSONL, daily rotation,
                        90-day retention. Reusable as-is.
  rag/embeddings.py     Ollama embedder (nomic-embed-text default),
                        load-balanced across N URLs. Reusable.
  rag/chunk.py          Paragraph-aware chunker with synthetic
                        chunk 0. Per-product tunable.
  rag/index.py          Chroma + BM25 builder. --rebuild and
                        --bm25-only flags.
  rag/bm25.py           SQLite FTS5 lexical index. Reusable.
  scrape/changelog.py   --cached / --ref / --json / --history-out.
                        Reusable.
  scrape/README.md      What you write per-product.
  eval/queries.jsonl.example
                        Curate ~25 hand-labeled queries here.
  eval/retrievers.py    Retriever protocol + stub classes.
  eval/run_eval.py      MRR / Recall@K / nDCG@K harness skeleton.
  scripts/usage_report.py
                        Standalone log analyzer; the
                        FOLLOW-UP CHECKS pattern noted in the
                        module docstring.
  scripts/registry_gc.py
                        Gitea container registry cleanup. Reusable.

Deployment + CI:

  Dockerfile               Python 3.12-slim; COPY corpus + chroma
                           + bm25 last for cache efficiency.
  deploy/docker-compose.yml MCP + reranker sidecar + Watchtower.
                           Templated with <placeholders>.
  .gitea/workflows/refresh.yml    Weekly cron + manual dispatch.
                                  fetch-depth: 0, retry-on-race,
                                  three-tag image scheme.
  .gitea/workflows/image-only.yml Code-only ship cycle, ~18min.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.gitea/workflows/image-only.yml

@@ -0,0 +1,89 @@
+name: Image rebuild (skip scrape)
+
+# Fast path for code-only changes. Skips the scrape and goes straight to:
+# rebuild indexes (from corpus already committed on main) + image build
+# + push. Runtime is ~18 min vs ~40 min for the full refresh.
+#
+# Use when a PR only changes code/config — anything where the upstream
+# corpus hasn't moved but we want the new Python in the running image.
+#
+# IMPORTANT: fetch-depth: 0 is required for the digest-history step
+# to find commits to walk. Don't change to 1.
+
+on:
+  workflow_dispatch:
+
+env:
+  REGISTRY_PUSH: <lan-host>:<port>
+  REGISTRY_PULL: <public-registry-hostname>
+  IMAGE: <owner>/<product>-docs-mcp
+  OLLAMA_URL: http://<gpu-host>:11434
+  EMBED_MODEL: nomic-embed-text
+  PRODUCT_NAME: <product>
+
+jobs:
+  build:
+    runs-on: docker
+    container:
+      image: catthehacker/ubuntu:act-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          # Full history (not shallow) so the digest-history step can
+          # walk git log up to --history-days back.
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install -q --upgrade pip
+          python -m pip install -q -r requirements.txt
+
+      - name: Refresh digest history
+        # Cheap (a few seconds); doesn't touch corpus content.
+        # Without this step, a code-only deploy would ship an
+        # increasingly-stale digest history relative to git.
+        run: |
+          mkdir -p corpus/.digest
+          python -m scrape.changelog \
+              --history-out corpus/.digest/history.jsonl \
+              --history-days 120
+
+      - name: Verify committed corpus is present
+        run: |
+          test -d corpus || { echo "ERROR: corpus/ missing on this ref"; exit 1; }
+          echo "corpus: $(du -sh corpus | cut -f1), $(find corpus -name '*.md' | wc -l) markdown files"
+
+      - name: Rebuild indexes from existing corpus
+        run: python -m rag.index --rebuild
+
+      - name: Log in to registry (LAN endpoint)
+        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "${REGISTRY_PUSH}" -u <user> --password-stdin
+
+      - name: Build & push image
+        run: |
+          SHA_TAG=$(echo "$GITHUB_SHA" | cut -c1-12)
+          DATE_TAG=$(date -u +%Y.%m.%d)
+          docker build \
+            -t "${REGISTRY_PUSH}/${IMAGE}:latest" \
+            -t "${REGISTRY_PUSH}/${IMAGE}:${SHA_TAG}" \
+            -t "${REGISTRY_PUSH}/${IMAGE}:${DATE_TAG}" \
+            .
+          docker push "${REGISTRY_PUSH}/${IMAGE}:latest"
+          docker push "${REGISTRY_PUSH}/${IMAGE}:${SHA_TAG}"
+          docker push "${REGISTRY_PUSH}/${IMAGE}:${DATE_TAG}"
+
+      - name: Prune old container versions
+        env:
+          GITEA_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
+        run: |
+          python scripts/registry_gc.py \
+            --owner <user> \
+            --package <product>-docs-mcp \
+            --keep-days 90 \
+            --keep-latest 5

.gitea/workflows/refresh.yml

@@ -0,0 +1,158 @@
+name: Weekly docs refresh
+
+# Runs the full pipeline: scrape upstream → rebuild indexes → push
+# image. Cron'd weekly (Mondays). Skip the reindex + image-push if the
+# scrape produced no diff against the committed corpus.
+#
+# IMPORTANT: actions/checkout@v4 fetch-depth: 0 is required because
+# the digest-history step walks git log up to --history-days back.
+# With a shallow checkout the history file ships empty.
+
+on:
+  schedule:
+    - cron: "0 6 * * 1"     # Mondays 06:00 UTC
+  workflow_dispatch:
+    inputs:
+      force_build:
+        description: "Rebuild indexes + push image even if corpus is unchanged"
+        type: boolean
+        default: false
+
+env:
+  # If your registry sits behind Cloudflare with its 100 MB body cap,
+  # use a LAN endpoint for pushes (bypasses CF) and the public hostname
+  # for pulls (response bodies aren't capped).
+  REGISTRY_PUSH: <lan-host>:<port>
+  REGISTRY_PULL: <public-registry-hostname>
+  IMAGE: <owner>/<product>-docs-mcp
+
+  # Embedder. One URL per GPU; the indexer round-robins.
+  OLLAMA_URL: http://<gpu-host>:11434
+  EMBED_MODEL: nomic-embed-text
+
+  PRODUCT_NAME: <product>
+
+jobs:
+  refresh:
+    runs-on: docker
+    container:
+      image: catthehacker/ubuntu:act-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          # Full history — required for the digest-history step to
+          # walk git log. Default fetch-depth: 1 silently produces a
+          # 0-byte history file.
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install -q --upgrade pip
+          python -m pip install -q -r requirements.txt
+
+      # ---- Phase 1: scrape ---------------------------------------
+      - name: Refresh bundle catalog
+        run: python -m scrape.bundles
+
+      - name: Re-scrape all bundles
+        # --force re-fetches every page so we actually see upstream
+        # edits. Without it the runner skips pages already on disk.
+        run: python -m scrape.runner --all --force --concurrency 6
+
+      # ---- Build the digest history BEFORE committing ------------
+      # See PLAN.md Phase 13. Walks recent corpus-touching commits
+      # and writes corpus/.digest/history.jsonl. The current refresh
+      # gets added on the NEXT run's history (one-week lag is fine).
+      - name: Build digest history
+        run: |
+          mkdir -p corpus/.digest
+          python -m scrape.changelog \
+              --history-out corpus/.digest/history.jsonl \
+              --history-days 120
+
+      # ---- Commit + retry-on-race --------------------------------
+      - name: Commit corpus changes (if any)
+        id: commit
+        run: |
+          git config user.name "<product>-docs-refresh"
+          git config user.email "actions@<your-domain>"
+          git add bundles.json corpus
+          if git diff --cached --quiet; then
+            echo "no corpus changes — skipping reindex and image build"
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          echo "changed=true" >> "$GITHUB_OUTPUT"
+          python -m scrape.changelog --cached > /tmp/changelog.txt
+          summary=$(head -1 /tmp/changelog.txt)
+          ts=$(date -u +"%Y-%m-%dT%H:%MZ")
+          {
+            echo "weekly refresh: ${ts} — ${summary}"
+            echo ""
+            cat /tmp/changelog.txt
+          } > /tmp/commitmsg.txt
+          git commit -F /tmp/commitmsg.txt
+          # Retry on race: if main moved while we were scraping (a
+          # human merged a PR during the run), `git push` rejects
+          # with "fetch first". Rebase our corpus commit onto new
+          # main and retry. Corpus + code paths are disjoint, so
+          # the rebase is trivially clean.
+          attempt=1
+          while [ $attempt -le 3 ]; do
+            if git push; then
+              echo "pushed corpus changes (attempt $attempt)"
+              break
+            fi
+            if [ $attempt -eq 3 ]; then
+              echo "push still failing after 3 attempts — bailing"
+              exit 1
+            fi
+            git fetch origin main
+            git rebase origin/main || { echo "rebase conflict — bailing"; exit 1; }
+            attempt=$((attempt + 1))
+          done
+
+      # ---- Reindex Chroma + BM25 ---------------------------------
+      - name: Rebuild indexes
+        if: steps.commit.outputs.changed == 'true' || inputs.force_build == true
+        run: python -m rag.index --rebuild
+
+      # ---- Build & push image ------------------------------------
+      - name: Log in to registry (LAN endpoint)
+        if: steps.commit.outputs.changed == 'true' || inputs.force_build == true
+        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "${REGISTRY_PUSH}" -u <user> --password-stdin
+
+      - name: Build & push image
+        if: steps.commit.outputs.changed == 'true' || inputs.force_build == true
+        # Runner shell is /bin/sh — use cut instead of ${VAR::N}.
+        # Three tags: :latest (Watchtower target), :<sha12>
+        # (rollback pin), :<YYYY.MM.DD> (human-readable).
+        run: |
+          SHA_TAG=$(echo "$GITHUB_SHA" | cut -c1-12)
+          DATE_TAG=$(date -u +%Y.%m.%d)
+          docker build \
+            -t "${REGISTRY_PUSH}/${IMAGE}:latest" \
+            -t "${REGISTRY_PUSH}/${IMAGE}:${SHA_TAG}" \
+            -t "${REGISTRY_PUSH}/${IMAGE}:${DATE_TAG}" \
+            .
+          docker push "${REGISTRY_PUSH}/${IMAGE}:latest"
+          docker push "${REGISTRY_PUSH}/${IMAGE}:${SHA_TAG}"
+          docker push "${REGISTRY_PUSH}/${IMAGE}:${DATE_TAG}"
+
+      # ---- Registry GC -------------------------------------------
+      - name: Prune old container versions
+        if: steps.commit.outputs.changed == 'true' || inputs.force_build == true
+        env:
+          GITEA_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
+        run: |
+          python scripts/registry_gc.py \
+            --owner <user> \
+            --package <product>-docs-mcp \
+            --keep-days 90 \
+            --keep-latest 5