Model providers: OpenAI/xAI/Ollama + run several at once (registry)
Extends the #215 abstraction: - OpenAICompatibleLLMProvider / OpenAICompatibleEmbeddingProvider — one impl (via the official openai SDK) covers OpenAI, xAI (api.x.ai/v1), Ollama (…:11434/v1), OpenRouter, etc.; they differ only by base_url, key, and model. - Registry factory: build_llm_providers() / build_embedding_providers() return every provider whose credentials are configured, so you can run several concurrently. get_llm_provider(name)/get_embedding_provider(name) select by name, falling back to default_*_provider, then Null. - Per-provider env config (ANTHROPIC_*, OPENAI_*, XAI_*, OLLAMA_*) + DEFAULT_LLM_PROVIDER / DEFAULT_EMBEDDING_PROVIDER; documented in .env.example. Defaults keep AI off (empty registry). Embeddings now have real backends (OpenAI/Ollama), still separate from the LLM since Anthropic offers no embeddings endpoint. Tests cover multi-provider selection, default resolution, disabled-without-credentials, and null fail-loud. Full suite 87 passed. Relates to #215. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Justin Paul <justin@jpaul.me>
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
"""OpenAI-compatible providers (one implementation, many vendors).
|
||||
|
||||
OpenAI, xAI (api.x.ai/v1), Ollama (…:11434/v1), OpenRouter, Together, vLLM, etc.
|
||||
all speak the OpenAI Chat Completions / Embeddings API — they differ only by
|
||||
base URL, key, and model name. So a single class, parameterized by those, plugs
|
||||
in every one of them via the official `openai` SDK.
|
||||
"""
|
||||
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from app.integrations.models.base import EmbeddingProvider, LLMProvider
|
||||
|
||||
|
||||
class OpenAICompatibleLLMProvider(LLMProvider):
|
||||
def __init__(self, *, api_key: str | None, base_url: str, model: str, max_tokens: int = 4096) -> None:
|
||||
# Local backends (Ollama) ignore the key but the SDK requires a non-empty one.
|
||||
self._client = AsyncOpenAI(api_key=api_key or "not-needed", base_url=base_url)
|
||||
self._model = model
|
||||
self._max_tokens = max_tokens
|
||||
|
||||
async def complete(self, *, prompt: str, system: str | None = None) -> str:
|
||||
messages: list[dict] = []
|
||||
if system:
|
||||
messages.append({"role": "system", "content": system})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
resp = await self._client.chat.completions.create(
|
||||
model=self._model, max_tokens=self._max_tokens, messages=messages
|
||||
)
|
||||
return resp.choices[0].message.content or ""
|
||||
|
||||
|
||||
class OpenAICompatibleEmbeddingProvider(EmbeddingProvider):
|
||||
def __init__(self, *, api_key: str | None, base_url: str, model: str, dimensions: int) -> None:
|
||||
self._client = AsyncOpenAI(api_key=api_key or "not-needed", base_url=base_url)
|
||||
self._model = model
|
||||
self.dimensions = dimensions
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||
resp = await self._client.embeddings.create(model=self._model, input=texts)
|
||||
return [d.embedding for d in resp.data]
|
||||
Reference in New Issue
Block a user