justin/hvm-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: justin/hvm-docs:embed-pool-resilience
Branches Tags
justin/hvm-docs:main justin/hvm-docs:eval-baseline-20260610 justin/hvm-docs:embed-pool-resilience
..
pull from: justin/hvm-docs:eval-baseline-20260610
Branches Tags
justin/hvm-docs:main justin/hvm-docs:eval-baseline-20260610 justin/hvm-docs:embed-pool-resilience

1 Commits
embed-pool-resilience .. eval-baseline-20260610

Author SHA1 Message Date
justin 2e10279beb eval: new baseline on the 4-endpoint embed pool index 
22 queries against the prod image index rebuilt today on the expanded
GPU pool with the resilient embedder (PR #8): dense MRR 0.539→0.924,
bm25+rerank 0.920→0.959, hybrid_rrf+rerank 0.875→0.960 vs the
2026-05-22 baseline. No regression from mixed-provenance embeddings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 20:38:23 -04:00
4 changed files with 150 additions and 40 deletions
Expand all files Collapse all files
.gitea/workflows/image-only.yml
+1 -1
View File
@@ -22,7 +22,7 @@ env:
# Two GPU-pinned Ollama containers on the Gitea host — same infra
# zerto-docs uses. :11435 = Titan X, :11436 = 1080 Ti. Indexer
# round-robins per batch.
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436,http://192.168.0.125:11434,http://192.168.0.126:11434
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436
EMBED_MODEL: nomic-embed-text
PRODUCT_NAME: hvm
.gitea/workflows/refresh.yml
+1 -1
View File
@@ -34,7 +34,7 @@ env:
# :11435 owns the Titan X, :11436 owns the 1080 Ti; the indexer
# round-robins per batch so both cards run in parallel. The host's
# primary Ollama on :11434 is left alone for OpenWebUI etc.
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436,http://192.168.0.125:11434,http://192.168.0.126:11434
OLLAMA_URLS: http://192.168.0.2:11435,http://192.168.0.2:11436
EMBED_MODEL: nomic-embed-text
PRODUCT_NAME: hvm
eval/results/baseline-20260610.md
+141
View File
@@ -0,0 +1,141 @@
<!--
Baseline refresh — 2026-06-10.
First eval after the embed pool grew from 2 ripper Ollamas to 4 endpoints
(+ .0.125 RTX 4090, + .0.126) and rag/embeddings.py gained endpoint
rotation/split resilience (PR #8, port of zerto-docs PR #45). Run against
the chroma/bm25 indexes extracted from the prod image built by today's
image-only run (post-merge rebuild on the 4-endpoint pool); query embeds
via host Ollama :11434, reranker = the prod jina-rerank instance.
vs the 2026-05-22 baseline.md: bm25+rerank MRR 0.920→0.959,
hybrid_rrf+rerank 0.875→0.960 — no regression from the mixed-provenance
embeddings (zerto-docs ran a same-corpus ripper-only control to verify
this property of the shared pool; see zerto eval/results/control-20260610.md).
-->
# Retrieval eval — k=5
_22 hand-curated queries, generated 2026-06-10 20:36:50_
| Retriever | MRR | Recall@5 | nDCG@5 | avg latency |
| --- | ---: | ---: | ---: | ---: |
| `dense` | 0.924 | 0.894 | 0.892 | 181ms |
| `bm25` | 0.880 | 0.909 | 0.883 | 5ms |
| `hybrid_rrf` | 0.888 | 0.894 | 0.876 | 112ms |
| `bm25+rerank` | 0.959 | 0.955 | 0.955 | 152ms |
| `hybrid_rrf+rerank` | 0.960 | 0.939 | 0.944 | 255ms |
## Per-query results
| Retriever | Query | MRR | top-1 |
| --- | --- | ---: | --- |
| `dense` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
| `dense` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
| `dense` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
| `dense` | Japanese keyboard layout in console sessions | 0.000 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
| `dense` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
| `dense` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
| `dense` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
| `dense` | configure storage buckets | 0.333 | `hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-...` |
| `dense` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
| `dense` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
| `dense` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
| `dense` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
| `dense` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
| `dense` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
| `dense` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
| `dense` | create a user account | 1.000 | `hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-...` |
| `dense` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_2/GUID-0D60567D-2DD3-4D8B-...` |
| `dense` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
| `dense` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
| `dense` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
| `dense` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
| `dense` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
| `bm25` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
| `bm25` | create an instance backup | 1.000 | `hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-...` |
| `bm25` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
| `bm25` | Japanese keyboard layout in console sessions | 0.250 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
| `bm25` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
| `bm25` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
| `bm25` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_0/GUID-2686BF33-E793-4BF9-...` |
| `bm25` | configure storage buckets | 0.100 | `hvm_user_manual_8_1_0/GUID-6B35AE72-3D19-447A-...` |
| `bm25` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
| `bm25` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
| `bm25` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
| `bm25` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
| `bm25` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
| `bm25` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
| `bm25` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
| `bm25` | create a user account | 0.000 | `hvm_user_manual_8_1_0/GUID-83AC77E8-48AE-420C-...` |
| `bm25` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
| `bm25` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
| `bm25` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
| `bm25` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
| `bm25` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
| `bm25` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
| `hybrid_rrf` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
| `hybrid_rrf` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
| `hybrid_rrf` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
| `hybrid_rrf` | Japanese keyboard layout in console sessions | 0.000 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
| `hybrid_rrf` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_0/GUID-ECCA4FDD-37C8-45CE-...` |
| `hybrid_rrf` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
| `hybrid_rrf` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
| `hybrid_rrf` | configure storage buckets | 0.333 | `hvm_user_manual_8_1_0/GUID-35B83D2C-52C6-4240-...` |
| `hybrid_rrf` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_0/GUID-0CBD386D-DED9-474D-...` |
| `hybrid_rrf` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
| `hybrid_rrf` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
| `hybrid_rrf` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
| `hybrid_rrf` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
| `hybrid_rrf` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
| `hybrid_rrf` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
| `hybrid_rrf` | create a user account | 0.200 | `hvm_user_manual_8_1_2/GUID-83AC77E8-48AE-420C-...` |
| `hybrid_rrf` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
| `hybrid_rrf` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
| `hybrid_rrf` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
| `hybrid_rrf` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
| `hybrid_rrf` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
| `hybrid_rrf` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
| `bm25+rerank` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
| `bm25+rerank` | create an instance backup | 1.000 | `hvm_user_manual_8_1_1/GUID-9DA38943-BF95-446D-...` |
| `bm25+rerank` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
| `bm25+rerank` | Japanese keyboard layout in console sessions | 0.100 | `hvm_user_manual_8_1_0/GUID-BCD3CA0B-5C7E-46CB-...` |
| `bm25+rerank` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-...` |
| `bm25+rerank` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
| `bm25+rerank` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_1/GUID-2686BF33-E793-4BF9-...` |
| `bm25+rerank` | configure storage buckets | 1.000 | `hvm_user_manual_8_1_2/GUID-30F59A1F-0573-4D88-...` |
| `bm25+rerank` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_2/GUID-0CBD386D-DED9-474D-...` |
| `bm25+rerank` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
| `bm25+rerank` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
| `bm25+rerank` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
| `bm25+rerank` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
| `bm25+rerank` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
| `bm25+rerank` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
| `bm25+rerank` | create a user account | 1.000 | `hvm_user_manual_8_1_2/GUID-21972435-BFD0-481F-...` |
| `bm25+rerank` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_0/GUID-0D60567D-2DD3-4D8B-...` |
| `bm25+rerank` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
| `bm25+rerank` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
| `bm25+rerank` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
| `bm25+rerank` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
| `bm25+rerank` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
| `hybrid_rrf+rerank` | VME Manager sizing recommendations small medium large | 1.000 | `hvm_deployment_guide/GUID-0F55384D-5632-4CDC-...` |
| `hybrid_rrf+rerank` | create an instance backup | 1.000 | `hvm_user_manual_8_1_0/GUID-9DA38943-BF95-446D-...` |
| `hybrid_rrf+rerank` | what are the host hardware requirements | 1.000 | `hvm_deployment_guide/GUID-BE7493B3-B866-4269-...` |
| `hybrid_rrf+rerank` | Japanese keyboard layout in console sessions | 0.125 | `hvm_user_manual_8_1_2/GUID-BCD3CA0B-5C7E-46CB-...` |
| `hybrid_rrf+rerank` | elevate to Morpheus Enterprise | 1.000 | `hvm_user_manual_8_1_2/GUID-ECCA4FDD-37C8-45CE-...` |
| `hybrid_rrf+rerank` | create an HVM cluster | 1.000 | `hvm_deployment_guide/GUID-99397996-8315-49C1-...` |
| `hybrid_rrf+rerank` | back up and restore the VM Essentials manager | 1.000 | `hvm_user_manual_8_1_2/GUID-2686BF33-E793-4BF9-...` |
| `hybrid_rrf+rerank` | configure storage buckets | 1.000 | `hvm_user_manual_8_1_1/GUID-30F59A1F-0573-4D88-...` |
| `hybrid_rrf+rerank` | disable two-factor authentication | 1.000 | `hvm_user_manual_8_1_1/GUID-0CBD386D-DED9-474D-...` |
| `hybrid_rrf+rerank` | upgrading the manager | 1.000 | `hvm_deployment_guide/GUID-4EDB4324-2C3B-435F-...` |
| `hybrid_rrf+rerank` | supported storage protocols | 1.000 | `hvm_deployment_guide/GUID-6EBCB223-4C48-456F-...` |
| `hybrid_rrf+rerank` | network bonding configuration | 1.000 | `hvm_deployment_guide/GUID-6F1B4C62-CCE2-4AE5-...` |
| `hybrid_rrf+rerank` | what TCP ports does HVM need open | 1.000 | `hvm_deployment_guide/GUID-97DDED8D-EE6B-4819-...` |
| `hybrid_rrf+rerank` | install HVM OS on a host server | 1.000 | `hvm_deployment_guide/GUID-28F18596-4902-4CD1-...` |
| `hybrid_rrf+rerank` | configure Linux images for HVM clusters | 1.000 | `hvm_deployment_guide/GUID-8D494112-C361-4300-...` |
| `hybrid_rrf+rerank` | create a user account | 1.000 | `hvm_user_manual_8_1_1/GUID-21972435-BFD0-481F-...` |
| `hybrid_rrf+rerank` | Openstack Swift bucket | 1.000 | `hvm_user_manual_8_1_1/GUID-0D60567D-2DD3-4D8B-...` |
| `hybrid_rrf+rerank` | API reference for VM Essentials | 1.000 | `hvm_deployment_guide/GUID-88494051-74D1-4BD9-...` |
| `hybrid_rrf+rerank` | Worker version compatibility 8.1.2 | 1.000 | `hvm_release_notes_8_1_2/sd00007734en_us...` |
| `hybrid_rrf+rerank` | recommended converged networking setup scenario | 1.000 | `hvm_deployment_guide/GUID-2DD9D39D-9031-4BB5-...` |
| `hybrid_rrf+rerank` | qualification matrix supported hardware | 1.000 | `hvm_deployment_guide/GUID-E3635F0A-11DA-4078-...` |
| `hybrid_rrf+rerank` | configure the VM Essentials manager initial setup | 1.000 | `hvm_deployment_guide/GUID-456E190C-E912-4079-...` |
rag/embeddings.py
+7 -38
View File
@@ -19,7 +19,6 @@ from __future__ import annotations
import os
import logging
import time
from typing import Any
import httpx
@@ -44,18 +43,11 @@ EMBED_DIM = int(os.environ.get("EMBED_DIM", "768"))
class OllamaEmbeddings(EmbeddingFunction):
"""Calls /api/embed across N Ollama endpoints, round-robin per batch.
"""Calls /api/embed across N Ollama endpoints, naive round-robin.
For indexing throughput on multiple GPUs, run one Ollama container
per GPU (pinned via NVIDIA_VISIBLE_DEVICES) and pass all their URLs
in OLLAMA_URL — the embedder picks the next endpoint per batch.
Resilient (ported from zerto-docs PR #45): a failed call rotates to
the next endpoint and retries with backoff instead of failing the
whole rebuild. HTTP status errors additionally halve the input —
the .0.125 Windows Ollama (4090) 400s when its model runner dies on
an oversized input array, and one endpoint rejecting a batch the
others accept shouldn't kill a multi-hour index build.
"""
def __init__(self, urls: list[str] = OLLAMA_URLS, model: str = EMBED_MODEL):
@@ -64,37 +56,14 @@ class OllamaEmbeddings(EmbeddingFunction):
self._next = 0
def __call__(self, input: Documents) -> Embeddings:
return self._embed(list(input), attempt=1)
def _embed(self, texts: list, attempt: int) -> Embeddings:
url = self.urls[self._next % len(self.urls)]
self._next += 1
try:
with httpx.Client(timeout=300) as c:
r = c.post(f"{url}/api/embed",
json={"model": self.model, "input": texts})
r.raise_for_status()
return r.json().get("embeddings") or []
except (httpx.TransportError, httpx.HTTPStatusError) as e:
if isinstance(e, httpx.HTTPStatusError):
desc = f"HTTP {e.response.status_code} ({e.response.text[:200]})"
else:
desc = f"transport error {type(e).__name__}"
if attempt >= 5:
log.error("%s from %s (%d texts) — giving up after %d attempts",
desc, url, len(texts), attempt)
raise
if isinstance(e, httpx.HTTPStatusError) and len(texts) > 16:
mid = len(texts) // 2
log.warning("%s from %s — splitting %d texts into %d+%d (attempt %d)",
desc, url, len(texts), mid, len(texts) - mid, attempt)
return (self._embed(texts[:mid], attempt + 1)
+ self._embed(texts[mid:], attempt + 1))
backoff = 0.5 * (2 ** (attempt - 1)) # 0.5, 1, 2, 4
log.warning("%s (attempt %d, %s) — retrying in %.1fs",
desc, attempt, url, backoff)
time.sleep(backoff)
return self._embed(texts, attempt + 1)
with httpx.Client(timeout=300) as c:
r = c.post(f"{url}/api/embed",
json={"model": self.model, "input": list(input)})
r.raise_for_status()
data = r.json()
return data.get("embeddings") or []
def name(self) -> str: # newer chromadb requires this
return f"ollama:{self.model}"