142 lines
14 KiB
Markdown
142 lines
14 KiB
Markdown
<!--
|
|
Baseline refresh — 2026-06-10.
|
|
|
|
First eval after the embed pool grew from 2 ripper Ollamas to 4 endpoints
|
|
(+ .0.125 RTX 4090, + .0.126) and rag/embeddings.py gained endpoint
|
|
rotation/split resilience (PR #8, port of zerto-docs PR #45). Run against
|
|
the chroma/bm25 indexes extracted from the prod image built by today's
|
|
image-only run (post-merge rebuild on the 4-endpoint pool); query embeds
|
|
via host Ollama :11434, reranker = the prod jina-rerank instance.
|
|
|
|
vs the 2026-05-22 baseline.md: bm25+rerank MRR 0.920→0.959,
|
|
hybrid_rrf+rerank 0.875→0.960 — no regression from the mixed-provenance
|
|
embeddings (zerto-docs ran a same-corpus ripper-only control to verify
|
|
this property of the shared pool; see zerto eval/results/control-20260610.md).
|
|
-->
|
|
# Retrieval eval — k=5
|
|
|
|
_22 hand-curated queries, generated 2026-06-10 20:36:50_
|
|
|
|
| Retriever | MRR | Recall@5 | nDCG@5 | avg latency |
|
|
| --- | ---: | ---: | ---: | ---: |
|
|
| `dense` | 0.924 | 0.894 | 0.892 | 181ms |
|
|
| `bm25` | 0.880 | 0.909 | 0.883 | 5ms |
|
|
| `hybrid_rrf` | 0.888 | 0.894 | 0.876 | 112ms |
|
|
| `bm25+rerank` | 0.959 | 0.955 | 0.955 | 152ms |
|
|
| `hybrid_rrf+rerank` | 0.960 | 0.939 | 0.944 | 255ms |
|
|
|
|
## Per-query results
|
|
|
|
| Retriever | Query | MRR | top-1 |
|
|
| --- | --- | ---: | --- |
|
|
| `dense` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
|
|
| `dense` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
|
|
| `dense` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
|
|
| `dense` | Japanese keyboard layout in console sessions | 0.000 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
|
|
| `dense` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
|
|
| `dense` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
|
|
| `dense` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
|
|
| `dense` | configure storage buckets | 0.333 | `hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-...` |
|
|
| `dense` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
|
|
| `dense` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
|
|
| `dense` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
|
|
| `dense` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
|
|
| `dense` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
|
|
| `dense` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
|
|
| `dense` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
|
|
| `dense` | create a user account | 1.000 | `hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-...` |
|
|
| `dense` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_2/GUID-0D60567D-2DD3-4D8B-...` |
|
|
| `dense` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
|
|
| `dense` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
|
|
| `dense` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
|
|
| `dense` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
|
|
| `dense` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
|
|
| `bm25` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
|
|
| `bm25` | create an instance backup | 1.000 | `hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-...` |
|
|
| `bm25` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
|
|
| `bm25` | Japanese keyboard layout in console sessions | 0.250 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
|
|
| `bm25` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
|
|
| `bm25` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
|
|
| `bm25` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-...` |
|
|
| `bm25` | configure storage buckets | 0.100 | `hvm_user_manual_8_1_0/GUID-6B35AE72-3D19-447A-...` |
|
|
| `bm25` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
|
|
| `bm25` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
|
|
| `bm25` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
|
|
| `bm25` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
|
|
| `bm25` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
|
|
| `bm25` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
|
|
| `bm25` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
|
|
| `bm25` | create a user account | 0.000 | `hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-...` |
|
|
| `bm25` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
|
|
| `bm25` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
|
|
| `bm25` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
|
|
| `bm25` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
|
|
| `bm25` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
|
|
| `bm25` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
|
|
| `hybrid_rrf` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
|
|
| `hybrid_rrf` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
|
|
| `hybrid_rrf` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
|
|
| `hybrid_rrf` | Japanese keyboard layout in console sessions | 0.000 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
|
|
| `hybrid_rrf` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
|
|
| `hybrid_rrf` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
|
|
| `hybrid_rrf` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
|
|
| `hybrid_rrf` | configure storage buckets | 0.333 | `hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-...` |
|
|
| `hybrid_rrf` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
|
|
| `hybrid_rrf` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
|
|
| `hybrid_rrf` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
|
|
| `hybrid_rrf` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
|
|
| `hybrid_rrf` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
|
|
| `hybrid_rrf` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
|
|
| `hybrid_rrf` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
|
|
| `hybrid_rrf` | create a user account | 0.200 | `hvm_user_manual_8_1_2/GUID-83AC77E8-48AE-420C-...` |
|
|
| `hybrid_rrf` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
|
|
| `hybrid_rrf` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
|
|
| `hybrid_rrf` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
|
|
| `hybrid_rrf` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
|
|
| `hybrid_rrf` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
|
|
| `hybrid_rrf` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
|
|
| `bm25+rerank` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
|
|
| `bm25+rerank` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
|
|
| `bm25+rerank` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
|
|
| `bm25+rerank` | Japanese keyboard layout in console sessions | 0.100 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
|
|
| `bm25+rerank` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-...` |
|
|
| `bm25+rerank` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
|
|
| `bm25+rerank` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_1/GUID-2686BF33-E793-4BF9-...` |
|
|
| `bm25+rerank` | configure storage buckets | 1.000 | `hvm_user_manual_8_1_2/GUID-30F59A1F-0573-4D88-...` |
|
|
| `bm25+rerank` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_2/GUID-0CBD386D-DED9-474D-...` |
|
|
| `bm25+rerank` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
|
|
| `bm25+rerank` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
|
|
| `bm25+rerank` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
|
|
| `bm25+rerank` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
|
|
| `bm25+rerank` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
|
|
| `bm25+rerank` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
|
|
| `bm25+rerank` | create a user account | 1.000 | `hvm_user_manual_8_1_2/GUID-21972435-BFD0-481F-...` |
|
|
| `bm25+rerank` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
|
|
| `bm25+rerank` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
|
|
| `bm25+rerank` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
|
|
| `bm25+rerank` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
|
|
| `bm25+rerank` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
|
|
| `bm25+rerank` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
|
|
| `hybrid_rrf+rerank` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
|
|
| `hybrid_rrf+rerank` | create an instance backup | 1.000 | `hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-...` |
|
|
| `hybrid_rrf+rerank` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
|
|
| `hybrid_rrf+rerank` | Japanese keyboard layout in console sessions | 0.125 | `hvm_user_manual_8_1_2/GUID-BCD3CA0B-5C7E-46CB-...` |
|
|
| `hybrid_rrf+rerank` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-...` |
|
|
| `hybrid_rrf+rerank` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
|
|
| `hybrid_rrf+rerank` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
|
|
| `hybrid_rrf+rerank` | configure storage buckets | 1.000 | `hvm_user_manual_8_1_1/GUID-30F59A1F-0573-4D88-...` |
|
|
| `hybrid_rrf+rerank` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_1/GUID-0CBD386D-DED9-474D-...` |
|
|
| `hybrid_rrf+rerank` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
|
|
| `hybrid_rrf+rerank` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
|
|
| `hybrid_rrf+rerank` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
|
|
| `hybrid_rrf+rerank` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
|
|
| `hybrid_rrf+rerank` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
|
|
| `hybrid_rrf+rerank` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
|
|
| `hybrid_rrf+rerank` | create a user account | 1.000 | `hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-...` |
|
|
| `hybrid_rrf+rerank` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_1/GUID-0D60567D-2DD3-4D8B-...` |
|
|
| `hybrid_rrf+rerank` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
|
|
| `hybrid_rrf+rerank` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
|
|
| `hybrid_rrf+rerank` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
|
|
| `hybrid_rrf+rerank` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
|
|
| `hybrid_rrf+rerank` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
|