claude
7dbec5389d
Post-prefix eval (22 queries, 2026-09-30): dense MRR 0.966 vs bm25+rerank 0.827. Watchtower keeps live RERANK_URL, so rerank is gated on RERANK_ENABLED (default off) rather than URL presence.
872 B
872 B
Retrieval eval — k=5
22 hand-curated queries, generated 2026-09-30 02:40:07
Live container after nomic prefixes + heading-recursive chunking
(hvm-docs image b200c5f76c56, Watchtower 2026-09-30T02:36Z).
Compared with baseline.md (2026-05-22, pre-prefix).
| Retriever | P@1 | MRR | Recall@5 | nDCG@5 | avg latency |
|---|---|---|---|---|---|
dense |
0.955 | 0.966 | 0.985 | 0.972 | 183ms |
bm25 |
0.864 | 0.882 | 0.909 | 0.886 | 8ms |
hybrid_rrf |
0.955 | 0.961 | 0.955 | 0.955 | 113ms |
bm25+rerank |
0.773 | 0.827 | 0.848 | 0.816 | 162ms |
hybrid_rrf+rerank |
0.727 | 0.813 | 0.894 | 0.821 | 254ms |
Dense flipped from worst (May MRR 0.539) to best. Rerank now hurts.
search_docs therefore defaults to dense; rerank is opt-in via
RERANK_ENABLED.
Dense P@1 miss: create a user account (MRR 0.250).