Together AI has announced the launch of Together Compiler (TKCC), a groundbreaking system designed to optimize AI inference workloads through continuous compilation. The system, introduced on November 4, 2026, aims to address the challenges posed by the dynamic nature of AI workloads, which span diverse models, hardware platforms, and traffic patterns.

Addressing Variability in AI Inference

AI inference workloads are increasingly complex, with Together AI serving trillions of tokens daily across thousands of customers. These workloads vary significantly in batch size, sequence length, sparsity, and other factors. Additionally, the hardware landscape is highly fragmented, with platforms such as NVIDIA H100, B200, and Groq requiring distinct optimization strategies. Together Compiler leverages agent-driven optimization to adapt to these diverse conditions without relying on static rules or manual tuning.

Persistent Engines and Experiment Throughput

TKCC introduces "persistent engines" that eliminate the need for repeated engine startups during optimization. Traditional systems require up to 467 seconds for setup, while TKCC reduces this to just 9 seconds. This innovation enables a 252x increase in experiment throughput compared to baseline agentic systems. Agents within TKCC share optimization proposals via a message board, ensuring continuous testing and minimizing GPU idle time.

Performance Benchmarks

In testing, TKCC achieved up to 2.6x higher tokens per second (TPS) across several open-source models, including Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. It set new records on the SemiAnalysis InferenceX benchmark, delivering 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash. Additionally, TKCC outperformed open-source megakernels by 1.7x in Llama 1B batch size 1 settings.

Adapting to Traffic Drift

Together Compiler is designed to handle evolving user behavior. For example, prompt lengths increased by 80% within two weeks of a new model release, and regional differences in traffic patterns were observed, with some countries averaging 97,000 tokens per prompt compared to 70,000 in others. TKCC dynamically adjusts to these shifts, ensuring optimal performance without requiring customer code changes.

Broader Applications

Beyond language model inference, TKCC has demonstrated its ability to generalize to other domains, including weather modeling, partial differential equations (PDEs), time-series analysis, and biology. This versatility underscores its potential as a universal tool for optimizing computational workloads.

With its innovative architecture and significant performance improvements, Together Compiler represents a major step forward in AI inference optimization, enabling faster and more efficient deployment of AI models across diverse use cases.