Together AI Refines Model Deployment

Together AI is detailing its approach to dedicated model inference, a system designed for predictable and scalable AI model deployment. The platform hinges on a three-part structure: endpoints, deployments, and configurations, all orchestrated by a capacity-aware traffic split. This system, outlined in a recent technical post, allows for advanced deployment strategies like rollouts, A/B tests, and shadow experiments, ensuring zero-downtime updates. The core innovation lies in how traffic is routed, prioritizing actual capacity over fixed percentages, a concept crucial for efficient inference.

At the heart of the system is the 'config,' a immutable recipe defining the inference engine, GPU specifications, and optimization profiles (latency, throughput, or balanced). Each config is tied to a specific model revision in a 'deployment,' which includes autoscaling policies. Endpoints serve as stable, user-facing names that direct traffic across these deployments based on their assigned weights and ready replica counts. This layered approach provides granular control and robust management for complex AI workloads.

Capacity-Aware Routing Explained

The traffic split is key: a list of {deployment_id, weight} pairs. The system calculates each deployment's effective capacity by multiplying its weight by its number of ready replicas. Traffic is then routed proportionally to this calculated capacity. For instance, if two deployments share equal weights but one has more active replicas, it will naturally receive a larger share of the traffic. This design ensures that scaling one deployment automatically adjusts its traffic load, simplifying management and preventing idle resources.

This capacity-driven routing means that deployments with zero ready replicas, perhaps due to cold starts or failures, contribute zero capacity and receive no traffic. Traffic seamlessly flows to available deployments, ensuring continuous service. This contrasts with naive percentage-based splits, which can lead to saturation or underutilization as replica counts fluctuate.

The platform also offers flexibility through A/B testing cohorts, which allow for fixed traffic shares using integer percentages that sum to 100%. This is managed via simple API updates, enabling dynamic adjustments to traffic distribution without service interruptions.

Under the Hood: Choosing Your Config

Users don't need to craft configurations from scratch. Together AI provides pre-benchmarked 'deployment profiles' for supported models. These profiles are certified model-and-config pairs, continuously refined by the platform. For example, users can explore profiles for models like the Gemma 4 31B deployment, selecting hardware and optimization settings that best suit their needs.

When selecting a config, key selectors include accelerator type, GPU count, and optimization profile. The optimization axis is particularly critical. Latency profiles prioritize faster response times for interactive applications, while throughput profiles maximize token output for batch processing. A balanced profile serves as a versatile default for mixed workloads. The immutability of configs ensures that a deployment's behavior remains consistent, with changes requiring new revisions, thus providing a stable rollback target.

Experimenting with Dedicated Model Inference

Together AI demonstrated its capacity-aware routing with an experiment involving two single-H100 deployments. Initially, with one replica each and equal weights, traffic split 50/50. When one deployment scaled to two replicas, the traffic split shifted to approximately 66.7/33.3, precisely following the increased capacity. This real-world test validates the system's ability to dynamically manage load based on available resources.

The company also presented performance benchmarks comparing different optimization profiles for models like the MiniMax M3 inference. These benchmarks highlight the critical importance of measuring performance under varying concurrency levels, as a profile that excels at low concurrency might degrade significantly under higher loads. This underscores the value of dedicated model inference configurations tailored to specific traffic patterns.

The platform aims to make advanced inference capabilities accessible. With just a few commands, developers can deploy models, configure endpoints, and start serving traffic, leveraging the power of advanced infrastructure without deep operational overhead. This focus on ease of use, combined with sophisticated routing and scaling mechanisms, positions Together AI Dedicated Model Inference as a potent solution for deploying AI models at scale.