justin
dda044eb95
Phase 3/6/7/8 in one pass since they depend on each other.
* docs_mcp/server.py
- Wire search_docs / get_page / list_versions tool bodies.
- search_docs flow: BM25 first (rag.bm25 FTS5) → over-fetch RERANK_POOL
chunks → POST to RERANK_URL/v1/rerank → return top-k. Dense is the
fallback when BM25 finds nothing. HYBRID_SEARCH=true switches to
dense+BM25+RRF (fused via the new _rrf_fuse helper).
- All retrieval failures are caught and fall back to the next layer,
so a dead reranker or missing BM25 db never blocks a search.
- Source URLs built from the bundle's docId so results link straight
into support.hpe.com.
* eval/
- 22 hand-curated golden queries grounded in real corpus page titles.
- DenseRetriever / BM25Retriever / HybridRetriever / RerankedRetriever
+ MRR/Recall@K/nDCG@K harness. RERANK_URL env activates the
reranked variants.
- Committed eval/results/baseline.md. On this corpus:
dense: MRR 0.539
bm25: MRR 0.880
hybrid_rrf: MRR 0.692
bm25+rerank: MRR 0.920 (winner)
hybrid_rrf+rerank: MRR 0.875
HPE structured docs use controlled vocabulary, so lexical match
dominates. Hybrid loses because dense pollutes the fused pool.
* scripts/rerank_server.py
- Minimal HTTP /v1/rerank over sentence-transformers
cross-encoder/ms-marco-MiniLM-L-6-v2. Cohere-style request/response.
- This is the dev/CPU fallback; production replaces it with the
llama.cpp + jina-reranker-v2-base GGUF sidecar (same wire protocol).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
13 KiB
13 KiB
Retrieval eval — k=5
22 hand-curated queries, generated 2026-05-22 12:35:57
| Retriever | MRR | Recall@5 | nDCG@5 | avg latency |
|---|---|---|---|---|
dense |
0.539 | 0.621 | 0.558 | 88ms |
bm25 |
0.880 | 0.909 | 0.883 | 3ms |
hybrid_rrf |
0.692 | 0.818 | 0.713 | 69ms |
bm25+rerank |
0.920 | 0.939 | 0.927 | 490ms |
hybrid_rrf+rerank |
0.875 | 0.894 | 0.881 | 549ms |
Per-query results
| Retriever | Query | MRR | top-1 |
|---|---|---|---|
dense |
VME Manager sizing recommendations small medium large | 0.000 | hvm_user_manual_8_1_0/GUID-5D0F2D33-623B-4AA3-... |
dense |
create an instance backup | 0.250 | hvm_user_manual_8_1_0/GUID-1C1ADB18-710D-40A4-... |
dense |
what are the host hardware requirements | 0.500 | hvm_deployment_guide/GUID-3DA92E9D-0635-427A-... |
dense |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
dense |
elevate to Morpheus Enterprise | 0.000 | hvm_user_manual_8_1_0/GUID-37CA1665-B331-452A-... |
dense |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
dense |
back up and restore the VM Essentials manager | 0.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
dense |
configure storage buckets | 0.100 | hvm_user_manual_8_1_0/GUID-3AC9CEB6-F60C-46FA-... |
dense |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
dense |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
dense |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
dense |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
dense |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
dense |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
dense |
configure Linux images for HVM clusters | 0.000 | hvm_deployment_guide/GUID-EC50D35A-3B63-44DD-... |
dense |
create a user account | 0.000 | hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-... |
dense |
Openstack Swift bucket | 0.000 | hvm_user_manual_8_1_0/GUID-B9045AFD-6F61-42D7-... |
dense |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
dense |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
dense |
recommended converged networking setup scenario | 0.500 | hvm_deployment_guide/GUID-BDF3EFBF-EA90-4E6F-... |
dense |
qualification matrix supported hardware | 0.500 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
dense |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
bm25 |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
bm25 |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
bm25 |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
bm25 |
Japanese keyboard layout in console sessions | 0.250 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
bm25 |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
bm25 |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
bm25 |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-... |
bm25 |
configure storage buckets | 0.100 | hvm_user_manual_8_1_0/GUID-6B35AE72-3D19-447A-... |
bm25 |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
bm25 |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
bm25 |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
bm25 |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
bm25 |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
bm25 |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
bm25 |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
bm25 |
create a user account | 0.000 | hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-... |
bm25 |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
bm25 |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
bm25 |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
bm25 |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
bm25 |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
bm25 |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
hybrid_rrf |
VME Manager sizing recommendations small medium large | 0.200 | hvm_user_manual_8_1_0/GUID-BB3046E2-F2D4-4B45-... |
hybrid_rrf |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
hybrid_rrf |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
hybrid_rrf |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
hybrid_rrf |
elevate to Morpheus Enterprise | 0.500 | hvm_user_manual_8_1_0/GUID-37CA1665-B331-452A-... |
hybrid_rrf |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
hybrid_rrf |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-... |
hybrid_rrf |
configure storage buckets | 0.200 | hvm_user_manual_8_1_0/GUID-E32D92B2-8DB3-42C3-... |
hybrid_rrf |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
hybrid_rrf |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
hybrid_rrf |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
hybrid_rrf |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
hybrid_rrf |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
hybrid_rrf |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
hybrid_rrf |
configure Linux images for HVM clusters | 0.200 | hvm_deployment_guide/GUID-EC50D35A-3B63-44DD-... |
hybrid_rrf |
create a user account | 0.000 | hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-... |
hybrid_rrf |
Openstack Swift bucket | 0.125 | hvm_user_manual_8_1_0/GUID-6B35AE72-3D19-447A-... |
hybrid_rrf |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
hybrid_rrf |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
hybrid_rrf |
recommended converged networking setup scenario | 0.500 | hvm_deployment_guide/GUID-BDF3EFBF-EA90-4E6F-... |
hybrid_rrf |
qualification matrix supported hardware | 0.500 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
hybrid_rrf |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
bm25+rerank |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
bm25+rerank |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
bm25+rerank |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
bm25+rerank |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_2/GUID-BCD3CA0B-5C7E-46CB-... |
bm25+rerank |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
bm25+rerank |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
bm25+rerank |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-... |
bm25+rerank |
configure storage buckets | 0.250 | hvm_user_manual_8_1_0/GUID-E32D92B2-8DB3-42C3-... |
bm25+rerank |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
bm25+rerank |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
bm25+rerank |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
bm25+rerank |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
bm25+rerank |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
bm25+rerank |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
bm25+rerank |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
bm25+rerank |
create a user account | 1.000 | hvm_user_manual_8_1_0/GUID-21972435-BFD0-481F-... |
bm25+rerank |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
bm25+rerank |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
bm25+rerank |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
bm25+rerank |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
bm25+rerank |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
bm25+rerank |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
hybrid_rrf+rerank |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
hybrid_rrf+rerank |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
hybrid_rrf+rerank |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
hybrid_rrf+rerank |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
hybrid_rrf+rerank |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
hybrid_rrf+rerank |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
hybrid_rrf+rerank |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-... |
hybrid_rrf+rerank |
configure storage buckets | 0.250 | hvm_user_manual_8_1_0/GUID-E32D92B2-8DB3-42C3-... |
hybrid_rrf+rerank |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
hybrid_rrf+rerank |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
hybrid_rrf+rerank |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
hybrid_rrf+rerank |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
hybrid_rrf+rerank |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
hybrid_rrf+rerank |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
hybrid_rrf+rerank |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
hybrid_rrf+rerank |
create a user account | 0.000 | hvm_user_manual_8_1_0/GUID-443F9C0E-FFD9-48B9-... |
hybrid_rrf+rerank |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
hybrid_rrf+rerank |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
hybrid_rrf+rerank |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
hybrid_rrf+rerank |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
hybrid_rrf+rerank |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
hybrid_rrf+rerank |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |