The problem: workloads that change faster than compilation
Together AI describes in its arXiv contribution a recurring operational difficulty: AI workloads change rapidly because models are released more often, architectures are increasingly diverse, and traffic distribution changes frequently. Obtaining a compilation of a deployment into an optimized forward pass thus becomes a continuous problem, which the authors consider suited to an agent-guided optimization loop, a kind of automatic hill-climbing.
According to the material, existing systems struggle to perform continuous compilation because of a trade-off between measurement speed and measurement fidelity. Some use agents to optimize individual kernels of the forward pass with offline benchmarks, but the timings are unreliable because of data-dependent behavior in primitives such as mixture-of-experts (MoE). Other agents optimize the engine end-to-end directly, but incur slow startup times.
The technical proposal: a tiered engine and a cost model
To address this trade-off, Together AI presents Together Compiler, abbreviated TKCC, which it defines as the first true agent loop designed for continuous optimization of AI inference. The architecture is based on two elements: a Tiered Persistent Engine, which separates engine startup overhead from forward-pass computation, and a cost model that selects the parallelism configurations with which to start the optimization process.
The work also reports extensive ablations on agent context management and shares insights drawn from deploying TKCC in production. These last two aspects, however, are presented only as topics covered in the paper: the material provided does not detail their results.
The claimed numbers and what remains to be verified
The summary states that TKCC outperforms by up to 1.7 times hand-written, highly optimized Megakernels on the Llama 1B workload, described as a workhorse. On the InferenceX benchmark, compared with the best open-source engines, the system claims up to 2.6 times more tokens per second on the models described as the most recent: Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash, across several hardware platforms.
TKCC is also presented as state of the art at batch size 1, with 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash. The authors ultimately claim that the system generalizes beyond inference on frontier language models and that it serves trillions of tokens in the production fleet. These are the authors' claims, not results independently verified by the available material: benchmarks, hardware configurations and measurement method are not described in the summary.
