justin
de50f2c803
Extends the #215 abstraction: - OpenAICompatibleLLMProvider / OpenAICompatibleEmbeddingProvider — one impl (via the official openai SDK) covers OpenAI, xAI (api.x.ai/v1), Ollama (…:11434/v1), OpenRouter, etc.; they differ only by base_url, key, and model. - Registry factory: build_llm_providers() / build_embedding_providers() return every provider whose credentials are configured, so you can run several concurrently. get_llm_provider(name)/get_embedding_provider(name) select by name, falling back to default_*_provider, then Null. - Per-provider env config (ANTHROPIC_*, OPENAI_*, XAI_*, OLLAMA_*) + DEFAULT_LLM_PROVIDER / DEFAULT_EMBEDDING_PROVIDER; documented in .env.example. Defaults keep AI off (empty registry). Embeddings now have real backends (OpenAI/Ollama), still separate from the LLM since Anthropic offers no embeddings endpoint. Tests cover multi-provider selection, default resolution, disabled-without-credentials, and null fail-loud. Full suite 87 passed. Relates to #215. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Justin Paul <justin@jpaul.me>
41 lines
1.8 KiB
Python
41 lines
1.8 KiB
Python
"""OpenAI-compatible providers (one implementation, many vendors).
|
|
|
|
OpenAI, xAI (api.x.ai/v1), Ollama (…:11434/v1), OpenRouter, Together, vLLM, etc.
|
|
all speak the OpenAI Chat Completions / Embeddings API — they differ only by
|
|
base URL, key, and model name. So a single class, parameterized by those, plugs
|
|
in every one of them via the official `openai` SDK.
|
|
"""
|
|
|
|
from openai import AsyncOpenAI
|
|
|
|
from app.integrations.models.base import EmbeddingProvider, LLMProvider
|
|
|
|
|
|
class OpenAICompatibleLLMProvider(LLMProvider):
|
|
def __init__(self, *, api_key: str | None, base_url: str, model: str, max_tokens: int = 4096) -> None:
|
|
# Local backends (Ollama) ignore the key but the SDK requires a non-empty one.
|
|
self._client = AsyncOpenAI(api_key=api_key or "not-needed", base_url=base_url)
|
|
self._model = model
|
|
self._max_tokens = max_tokens
|
|
|
|
async def complete(self, *, prompt: str, system: str | None = None) -> str:
|
|
messages: list[dict] = []
|
|
if system:
|
|
messages.append({"role": "system", "content": system})
|
|
messages.append({"role": "user", "content": prompt})
|
|
resp = await self._client.chat.completions.create(
|
|
model=self._model, max_tokens=self._max_tokens, messages=messages
|
|
)
|
|
return resp.choices[0].message.content or ""
|
|
|
|
|
|
class OpenAICompatibleEmbeddingProvider(EmbeddingProvider):
|
|
def __init__(self, *, api_key: str | None, base_url: str, model: str, dimensions: int) -> None:
|
|
self._client = AsyncOpenAI(api_key=api_key or "not-needed", base_url=base_url)
|
|
self._model = model
|
|
self.dimensions = dimensions
|
|
|
|
async def embed(self, texts: list[str]) -> list[list[float]]:
|
|
resp = await self._client.embeddings.create(model=self._model, input=texts)
|
|
return [d.embedding for d in resp.data]
|