justin/hvm-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3b9cb4f8747ffa89c83fc46b3dc85ef5cae74c84
hvm-docs/eval/queries.jsonl
T
justin dda044eb95 search: BM25-default + cross-encoder rerank, hybrid behind env gate 
Phase 3/6/7/8 in one pass since they depend on each other.

* docs_mcp/server.py
  - Wire search_docs / get_page / list_versions tool bodies.
  - search_docs flow: BM25 first (rag.bm25 FTS5) → over-fetch RERANK_POOL
    chunks → POST to RERANK_URL/v1/rerank → return top-k. Dense is the
    fallback when BM25 finds nothing. HYBRID_SEARCH=true switches to
    dense+BM25+RRF (fused via the new _rrf_fuse helper).
  - All retrieval failures are caught and fall back to the next layer,
    so a dead reranker or missing BM25 db never blocks a search.
  - Source URLs built from the bundle's docId so results link straight
    into support.hpe.com.

* eval/
  - 22 hand-curated golden queries grounded in real corpus page titles.
  - DenseRetriever / BM25Retriever / HybridRetriever / RerankedRetriever
    + MRR/Recall@K/nDCG@K harness. RERANK_URL env activates the
    reranked variants.
  - Committed eval/results/baseline.md. On this corpus:
        dense:                MRR 0.539
        bm25:                 MRR 0.880
        hybrid_rrf:           MRR 0.692
        bm25+rerank:          MRR 0.920  (winner)
        hybrid_rrf+rerank:    MRR 0.875
    HPE structured docs use controlled vocabulary, so lexical match
    dominates. Hybrid loses because dense pollutes the fused pool.

* scripts/rerank_server.py
  - Minimal HTTP /v1/rerank over sentence-transformers
    cross-encoder/ms-marco-MiniLM-L-6-v2. Cohere-style request/response.
  - This is the dev/CPU fallback; production replaces it with the
    llama.cpp + jina-reranker-v2-base GGUF sidecar (same wire protocol).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 13:06:51 -04:00

23 lines
5.4 KiB
JSON
Raw Blame History

{"query": "VME Manager sizing recommendations small medium large", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-0F55384D-5632-4CDC-AA39-A21C1C089AFA"}], "tags": ["deployment", "sizing", "keyword-heavy"]}
{"query": "create an instance backup", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-9DA38943-BF95-446D-AB09-32323160D51B"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-9DA38943-BF95-446D-AB09-32323160D51B"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-9DA38943-BF95-446D-AB09-32323160D51B"}], "tags": ["backups", "how-to"]}
{"query": "what are the host hardware requirements", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-BE7493B3-B866-4269-9C13-ABFCF84658F2"}], "tags": ["deployment", "prereqs"]}
{"query": "Japanese keyboard layout in console sessions", "expected": [{"bundle_id": "hvm_release_notes_8_1_2", "page_id": "sd00007734en_us"}], "tags": ["release-notes", "8.1.2", "localization"]}
{"query": "elevate to Morpheus Enterprise", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-ECCA4FDD-37C8-45CE-A71F-C6E73B3BA713"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-ECCA4FDD-37C8-45CE-A71F-C6E73B3BA713"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-ECCA4FDD-37C8-45CE-A71F-C6E73B3BA713"}], "tags": ["upgrade"]}
{"query": "create an HVM cluster", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-99397996-8315-49C1-9E2F-2EED51CE03F3"}], "tags": ["deployment", "cluster"]}
{"query": "back up and restore the VM Essentials manager", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-2686BF33-E793-4BF9-98BE-82F800ED03EB"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-2686BF33-E793-4BF9-98BE-82F800ED03EB"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-2686BF33-E793-4BF9-98BE-82F800ED03EB"}], "tags": ["backups", "disaster-recovery"]}
{"query": "configure storage buckets", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-30F59A1F-0573-4D88-A679-B5C5168BCE6D"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-30F59A1F-0573-4D88-A679-B5C5168BCE6D"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-30F59A1F-0573-4D88-A679-B5C5168BCE6D"}], "tags": ["storage"]}
{"query": "disable two-factor authentication", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-0CBD386D-DED9-474D-A9F3-587F58B2D22D"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-0CBD386D-DED9-474D-A9F3-587F58B2D22D"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-0CBD386D-DED9-474D-A9F3-587F58B2D22D"}], "tags": ["security", "auth"]}
{"query": "upgrading the manager", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-4EDB4324-2C3B-435F-80FF-F430D02A2FDA"}], "tags": ["upgrade"]}
{"query": "supported storage protocols", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-6EBCB223-4C48-456F-950C-C8ED5610A0F8"}], "tags": ["deployment", "storage"]}
{"query": "network bonding configuration", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-6F1B4C62-CCE2-4AE5-9CE7-83C407BFE290"}], "tags": ["networking"]}
{"query": "what TCP ports does HVM need open", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-97DDED8D-EE6B-4819-8080-E163FD533CAB"}], "tags": ["networking", "firewall"]}
{"query": "install HVM OS on a host server", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-28F18596-4902-4CD1-83F3-1411430C5534"}], "tags": ["deployment", "install"]}
{"query": "configure Linux images for HVM clusters", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-8D494112-C361-4300-B7BF-B4DFE06E871C"}], "tags": ["deployment", "images"]}
{"query": "create a user account", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-21972435-BFD0-481F-A3E6-A52B116989F3"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-21972435-BFD0-481F-A3E6-A52B116989F3"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-21972435-BFD0-481F-A3E6-A52B116989F3"}], "tags": ["admin", "users"]}
{"query": "Openstack Swift bucket", "expected": [{"bundle_id": "hvm_user_manual_8_1_2", "page_id": "GUID-0D60567D-2DD3-4D8B-92D6-6849C7D773EA"}, {"bundle_id": "hvm_user_manual_8_1_1", "page_id": "GUID-0D60567D-2DD3-4D8B-92D6-6849C7D773EA"}, {"bundle_id": "hvm_user_manual_8_1_0", "page_id": "GUID-0D60567D-2DD3-4D8B-92D6-6849C7D773EA"}], "tags": ["storage", "rare-token"]}
{"query": "API reference for VM Essentials", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-88494051-74D1-4BD9-BAFF-134A573FF77B"}], "tags": ["api"]}
{"query": "Worker version compatibility 8.1.2", "expected": [{"bundle_id": "hvm_release_notes_8_1_2", "page_id": "sd00007734en_us"}], "tags": ["release-notes", "8.1.2"]}
{"query": "recommended converged networking setup scenario", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-2DD9D39D-9031-4BB5-A4ED-A0179BEF5259"}], "tags": ["networking", "deployment"]}
{"query": "qualification matrix supported hardware", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-E3635F0A-11DA-4078-8C3A-8D4B75724849"}], "tags": ["deployment", "compatibility"]}
{"query": "configure the VM Essentials manager initial setup", "expected": [{"bundle_id": "hvm_deployment_guide", "page_id": "GUID-456E190C-E912-4079-A691-8D2368D63748"}], "tags": ["deployment", "configuration"]}
Reference in New Issue View Git Blame Copy Permalink