claude
7dbec5389d
Post-prefix eval (22 queries, 2026-09-30): dense MRR 0.966 vs bm25+rerank 0.827. Watchtower keeps live RERANK_URL, so rerank is gated on RERANK_ENABLED (default off) rather than URL presence.
22 lines
872 B
Markdown
22 lines
872 B
Markdown
# Retrieval eval — k=5
|
|
|
|
_22 hand-curated queries, generated 2026-09-30 02:40:07_
|
|
|
|
Live container after nomic prefixes + heading-recursive chunking
|
|
(`hvm-docs` image `b200c5f76c56`, Watchtower 2026-09-30T02:36Z).
|
|
Compared with [`baseline.md`](baseline.md) (2026-05-22, pre-prefix).
|
|
|
|
| Retriever | P@1 | MRR | Recall@5 | nDCG@5 | avg latency |
|
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
| `dense` | 0.955 | 0.966 | 0.985 | 0.972 | 183ms |
|
|
| `bm25` | 0.864 | 0.882 | 0.909 | 0.886 | 8ms |
|
|
| `hybrid_rrf` | 0.955 | 0.961 | 0.955 | 0.955 | 113ms |
|
|
| `bm25+rerank` | 0.773 | 0.827 | 0.848 | 0.816 | 162ms |
|
|
| `hybrid_rrf+rerank` | 0.727 | 0.813 | 0.894 | 0.821 | 254ms |
|
|
|
|
Dense flipped from worst (May MRR 0.539) to best. Rerank now hurts.
|
|
`search_docs` therefore defaults to dense; rerank is opt-in via
|
|
`RERANK_ENABLED`.
|
|
|
|
Dense P@1 miss: `create a user account` (MRR 0.250).
|