Otari: Own Your AI Stack

The AI landscape often presents a stark choice: opt for a polished, closed-source stack with integrated tools and controls, or embrace the flexibility and sovereignty of open-weight models, only to find yourself rebuilding much of that stack from scratch. Otari aims to close this gap with its new open-source LLM gateway and a corresponding hosted platform, Otari.ai. This move, detailed on the Mozilla Blog, promises a unified developer experience across any LLM, whether frontier or open-weight.

Frontier AI providers bundle everything from execution environments to spend controls. However, switching to an open-weight model typically strips away these conveniences, leaving developers to manage crucial functionalities like usage tracking, budget management, and team administration themselves. Otari addresses this by providing these missing pieces as part of its gateway.

Bridging the Capability Gap

Otari's core offering is to equip open-weight models with the advanced capabilities typically found only in closed-source offerings. This includes server-side, model-agnostic tools such as sandboxed code execution via a Python REPL, web search integration using options like SearXNG, and OpenAI-compatible endpoints for transcription and image generation. This ensures that multimodal pipelines and agentic workflows remain intact, regardless of the underlying model.

The platform also facilitates LLM-powered document reranking for RAG systems and supports OpenAI-compatible asynchronous batch processing for cost-sensitive workloads. This commitment to feature parity is crucial for developers who want the benefits of open-source models without sacrificing essential functionalities. Future updates plan to incorporate guardrails for local, fast execution, even without dedicated GPUs, enhancing LLM gateway features.

This initiative mirrors broader industry efforts to enhance AI governance, such as Databricks' addition of AI guardrails and their focus on AI agent governance.

The Operational Backbone

Beyond advanced capabilities, Otari delivers the essential operational infrastructure teams often build in-house. This includes virtual API keys for secure credential management, per-user spending caps, real-time usage and cost tracking, and configurable rate limiting. Prometheus metrics are available for monitoring health and performance.

Otari.ai: The Managed Service

For those who prefer not to self-host, Otari.ai offers a managed, team-oriented platform built on the open-source gateway. It provides identity and team management with role-based access, workspace scoping, and routing policies for managing requests across different models and providers. The service includes a secure vault for encrypted credentials and offers managed providers, allowing access to frontier models and first-party open-weight models without needing personal API keys.

Otari.ai features multi-level budgets, declarative configuration via YAML, and robust observability tools including OTLP trace ingestion and OpenSearch-backed analytics. This transparency and control are key selling points for organizations prioritizing privacy and cost management.

Open Core Philosophy

Otari operates on an open-core model, with Otari.ai serving as a transparent business layer. This approach allows users to self-host for maximum privacy, ensuring prompts, completions, and logs never leave their environment. Alternatively, they can leverage the hosted platform for velocity, with the assurance that the underlying engine and API surface remain consistent.

A central tenet of Otari's design is making open-weight models first-class citizens, ensuring they benefit from the same management tools and capabilities as proprietary models. This bet underpins both the open-source project and the commercial platform, aligning with Mozilla.ai’s broader initiatives.

Getting started is straightforward: sign up for Otari.ai or clone the Otari open-source repository and run it locally via Docker Compose. Feedback is actively sought through GitHub and community channels.