Together AI has introduced the Together Compiler (TKCC), a system designed to optimize AI inference workloads dynamically and continuously. Announced on November 4, 2026, the compiler leverages high-throughput agents and persistent engine technology to adapt to diverse models, hardware platforms, and traffic patterns, achieving significant performance gains across various benchmarks.

Addressing Dynamic AI Inference Workloads

AI inference workloads are increasingly complex, spanning thousands of models, hardware platforms, and traffic shapes. Together AI reports that its infrastructure serves trillions of tokens daily across thousands of customers, each with unique requirements such as batch size, sequence length, and attention sparsity. Traditional manual optimization methods have proven unsustainable in this environment.

TKCC addresses these challenges by employing agents to rewrite GPU code and optimize inference engines. Unlike traditional systems that require lengthy engine restarts for each experiment, TKCC uses persistent engines capable of hot-reloading candidate optimizations. This approach minimizes GPU idle time and increases experimentation throughput by 252 times compared to baseline agentic systems.

Performance Benchmarks and Achievements

In less than 24 hours, TKCC achieved throughput improvements of 1.3 to 3 times across several open-source models, including Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. Notably, it set state-of-the-art records on SemiAnalysis’s InferenceX benchmark for low-latency performance, reaching 1,111 tokens per second (TPS) on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash. Additionally, TKCC outperformed open-source megakernels by 1.7 times in the Llama 1B batch size 1 setting.

The system also generalizes beyond language model inference, demonstrating applicability in domains such as weather modeling, partial differential equations (PDE), time-series analysis, and biology. Together AI highlights the compiler’s ability to adapt to changing user behaviors, including regional traffic patterns, new model releases, and bursty workloads.

Innovative Architecture and Optimization Strategies

TKCC’s architecture incorporates several innovations:

  • Persistent Engines: These engines eliminate the need for repeated start-up processes, reducing total setup time from 467 seconds to just 9 seconds.
  • Two-Tiered Optimization: A combination of "fast engines" for rapid iteration and "slower engines" for high-fidelity testing balances throughput and measurement accuracy.
  • Custom Cost Model: This model selects parallelism strategies, such as tensor parallelism (TP) and expert parallelism (EP), based on hardware specifications, microbenchmarks, and real-world traffic data.

These features enable TKCC to optimize configurations under service-level agreement (SLA) requirements while accommodating data-dependent traffic properties like uneven prompt lengths and MoE expert imbalances.

Real-World Impact and Future Potential

Phonic, a customer of Together AI, reported a nearly 50% reduction in median time to first speech for its Phonic-1 model after overnight optimization using TKCC. Together AI states that the compiler is already operational, serving trillions of tokens daily without requiring additional customer effort.

By addressing the challenges of dynamic inference workloads with a scalable and adaptive approach, TKCC represents a significant advancement in AI system optimization. Its ability to deliver rapid, high-fidelity optimizations positions it as a critical tool for organizations operating in latency-sensitive and computationally intensive environments.