Otoroshi extension · Open source
Otoroshi LLM Extension
The open-source AI gateway for Otoroshi.
Connect, set up, secure and seamlessly manage LLMs through a universal, OpenAI-compatible API. 50+ providers, guardrails, semantic cache, budgets, MCP and AI Studio.
- Stars
- 19
- Language
- Scala
- License
- Apache-2.0
- Last update
- today
Features
What's inside
Unified, OpenAI-compatible API
One interface for 50+ providers, including sovereign European ones (Mistral, Scaleway, OVHcloud, Cloud Temple) and local models with Ollama.
Load balancing, fallbacks, retries
Distribute workloads across providers and rescue failing requests automatically.
Simple & semantic cache
Speed up repeated queries, improve response times and reduce costs.
Quotas, budgets & costs
Token quotas per consumer, budgets, cost tracking and ecological impact reporting.
Guardrails
Validate prompts and responses against PII and secrets leakage, prompt injection, toxicity, gibberish and more.
MCP, agents & workflows
MCP connectors, virtual MCP servers and registry, tool calling, persistent memories and agentic workflows.
Multi-modal
Audio (TTS, STT, translation), images, video, embeddings and vector stores through the same gateway.
Key vault & fine-grained authorizations
Keep provider keys in Otoroshi vaults and constrain model usage by identity, API key or any request detail.
AI Studio
A self-service AI console on top of the gateway, with workspaces, playground, logs and budgets.
- OpenAI
- Anthropic
- Mistral AI
- Gemini
- Ollama
- Azure AI
- Groq
- DeepSeek
- Scaleway
- OVHcloud
- Cohere
- Hugging Face
- Cloudflare AI
- ElevenLabs
- + many more, local models too
AI Studio
One API
for every model.
AI Studio is an OpenRouter-like console served by Otoroshi itself: workspaces, bring-your-own-key providers, API keys, a chat playground, activity, logs and budgets — all stored as plain Otoroshi entities, manageable through the admin API, Kubernetes or GitOps.
- Workspaces
- API keys & BYOK
- Chat playground
- Activity & logs
Guardrails
Safe prompts.
Safe answers.
Guardrails validate prompts and responses before anything reaches a model or a user: personal and sensitive information, secrets, prompt injection, toxic or irrelevant content. Chain them per route, per model or per workspace.
AI Studio Enterprise
Share AI Studio with your whole organization
Your company login, members and roles in every workspace, secrets kept on the server and an audit trail: the same studio, as a standalone application for every team.
More open-source projects
Run it in productionwithout running Otoroshi.
Our extensions are available on Otoroshi Managed, fully operated by the people that wrote them. Or get professional support for your own clusters.