justin
c33682c968
22 queries against the prod image index rebuilt today on the expanded GPU pool with the resilient embedder (PR #8): dense MRR 0.539→0.557, bm25+rerank 0.920→0.959, hybrid_rrf+rerank 0.875→0.960 vs the 2026-05-22 baseline. No regression from mixed-provenance embeddings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
14 KiB
14 KiB
Retrieval eval — k=5
22 hand-curated queries, generated 2026-06-10 20:36:50
| Retriever | MRR | Recall@5 | nDCG@5 | avg latency |
|---|---|---|---|---|
dense |
0.924 | 0.894 | 0.892 | 181ms |
bm25 |
0.880 | 0.909 | 0.883 | 5ms |
hybrid_rrf |
0.888 | 0.894 | 0.876 | 112ms |
bm25+rerank |
0.959 | 0.955 | 0.955 | 152ms |
hybrid_rrf+rerank |
0.960 | 0.939 | 0.944 | 255ms |
Per-query results
| Retriever | Query | MRR | top-1 |
|---|---|---|---|
dense |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
dense |
create an instance backup | 1.000 | hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-... |
dense |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
dense |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
dense |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
dense |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
dense |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-... |
dense |
configure storage buckets | 0.333 | hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-... |
dense |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
dense |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
dense |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
dense |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
dense |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
dense |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
dense |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
dense |
create a user account | 1.000 | hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-... |
dense |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_2/GUID-0D60567D-2DD3-4D8B-... |
dense |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
dense |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
dense |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
dense |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
dense |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
bm25 |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
bm25 |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
bm25 |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
bm25 |
Japanese keyboard layout in console sessions | 0.250 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
bm25 |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
bm25 |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
bm25 |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-... |
bm25 |
configure storage buckets | 0.100 | hvm_user_manual_8_1_0/GUID-6B35AE72-3D19-447A-... |
bm25 |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
bm25 |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
bm25 |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
bm25 |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
bm25 |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
bm25 |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
bm25 |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
bm25 |
create a user account | 0.000 | hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-... |
bm25 |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
bm25 |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
bm25 |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
bm25 |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
bm25 |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
bm25 |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
hybrid_rrf |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
hybrid_rrf |
create an instance backup | 1.000 | hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-... |
hybrid_rrf |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
hybrid_rrf |
Japanese keyboard layout in console sessions | 0.000 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
hybrid_rrf |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-... |
hybrid_rrf |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
hybrid_rrf |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-... |
hybrid_rrf |
configure storage buckets | 0.333 | hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-... |
hybrid_rrf |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-... |
hybrid_rrf |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
hybrid_rrf |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
hybrid_rrf |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
hybrid_rrf |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
hybrid_rrf |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
hybrid_rrf |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
hybrid_rrf |
create a user account | 0.200 | hvm_user_manual_8_1_2/GUID-83AC77E8-48AE-420C-... |
hybrid_rrf |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
hybrid_rrf |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
hybrid_rrf |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
hybrid_rrf |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
hybrid_rrf |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
hybrid_rrf |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
bm25+rerank |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
bm25+rerank |
create an instance backup | 1.000 | hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-... |
bm25+rerank |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
bm25+rerank |
Japanese keyboard layout in console sessions | 0.100 | hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-... |
bm25+rerank |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-... |
bm25+rerank |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
bm25+rerank |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_1/GUID-2686BF33-E793-4BF9-... |
bm25+rerank |
configure storage buckets | 1.000 | hvm_user_manual_8_1_2/GUID-30F59A1F-0573-4D88-... |
bm25+rerank |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_2/GUID-0CBD386D-DED9-474D-... |
bm25+rerank |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
bm25+rerank |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
bm25+rerank |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
bm25+rerank |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
bm25+rerank |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
bm25+rerank |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
bm25+rerank |
create a user account | 1.000 | hvm_user_manual_8_1_2/GUID-21972435-BFD0-481F-... |
bm25+rerank |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-... |
bm25+rerank |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
bm25+rerank |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
bm25+rerank |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
bm25+rerank |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
bm25+rerank |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |
hybrid_rrf+rerank |
VME Manager sizing recommendations small medium large | 1.000 | hvm_deployment_guide/GUID-0F55384D-5632-4CDC-... |
hybrid_rrf+rerank |
create an instance backup | 1.000 | hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-... |
hybrid_rrf+rerank |
what are the host hardware requirements | 1.000 | hvm_deployment_guide/GUID-BE7493B3-B866-4269-... |
hybrid_rrf+rerank |
Japanese keyboard layout in console sessions | 0.125 | hvm_user_manual_8_1_2/GUID-BCD3CA0B-5C7E-46CB-... |
hybrid_rrf+rerank |
elevate to Morpheus Enterprise | 1.000 | hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-... |
hybrid_rrf+rerank |
create an HVM cluster | 1.000 | hvm_deployment_guide/GUID-99397996-8315-49C1-... |
hybrid_rrf+rerank |
back up and restore the VM Essentials manager | 1.000 | hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-... |
hybrid_rrf+rerank |
configure storage buckets | 1.000 | hvm_user_manual_8_1_1/GUID-30F59A1F-0573-4D88-... |
hybrid_rrf+rerank |
disable two-factor authentication | 1.000 | hvm_user_manual_8_1_1/GUID-0CBD386D-DED9-474D-... |
hybrid_rrf+rerank |
upgrading the manager | 1.000 | hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-... |
hybrid_rrf+rerank |
supported storage protocols | 1.000 | hvm_deployment_guide/GUID-6EBCB223-4C48-456F-... |
hybrid_rrf+rerank |
network bonding configuration | 1.000 | hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-... |
hybrid_rrf+rerank |
what TCP ports does HVM need open | 1.000 | hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-... |
hybrid_rrf+rerank |
install HVM OS on a host server | 1.000 | hvm_deployment_guide/GUID-28F18596-4902-4CD1-... |
hybrid_rrf+rerank |
configure Linux images for HVM clusters | 1.000 | hvm_deployment_guide/GUID-8D494112-C361-4300-... |
hybrid_rrf+rerank |
create a user account | 1.000 | hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-... |
hybrid_rrf+rerank |
Openstack Swift bucket | 1.000 | hvm_user_manual_8_1_1/GUID-0D60567D-2DD3-4D8B-... |
hybrid_rrf+rerank |
API reference for VM Essentials | 1.000 | hvm_deployment_guide/GUID-88494051-74D1-4BD9-... |
hybrid_rrf+rerank |
Worker version compatibility 8.1.2 | 1.000 | hvm_release_notes_8_1_2/sd00007734en_us... |
hybrid_rrf+rerank |
recommended converged networking setup scenario | 1.000 | hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-... |
hybrid_rrf+rerank |
qualification matrix supported hardware | 1.000 | hvm_deployment_guide/GUID-E3635F0A-11DA-4078-... |
hybrid_rrf+rerank |
configure the VM Essentials manager initial setup | 1.000 | hvm_deployment_guide/GUID-456E190C-E912-4079-... |