What Zyphra announced
Zyphra has published ZYFA (Zyphra FlashAttention on AMD), a kernel library for prefill-type attention in FP8 and BF16 precision aimed at AMD Instinct MI355X GPUs, as reported by MarkTechPost. The code is distributed under the Apache 2.0 license and is accompanied by a technical article. According to the company, ZYFA powers the attention engine of the Zyphra Optimization Engine, the system that manages the Zyphra Cloud service, and is presented as a drop-in replacement interface for flash_attn_func.
The release concerns a specific phase of the computation, the prefill of causal attention, with head size 128 and the shape required by the MLA mechanism (192/128). The stated environment is narrow: MI355X GPUs with CDNA4 architecture (gfx950), ROCm 7.2.3 and PyTorch 2.12. The BF16 kernels are written in HIP, while the FP8 kernels are manually scheduled assembly generated by a Python script.
The claimed figures and the comparison with AITER
The comparison point chosen by Zyphra is AITER, AMD's default kernel library. On real model activations, ZYFA's FP8 kernels are 1.16 to 1.76 times faster: 1.16x on Mistral-7B, 1.17x on Qwen3-8B and 1.76x on DeepSeek-V2-Lite, measured at 16K tokens and with sustained clocks. In BF16 the gain is marginal, between 1.01 and 1.02 times. In Gaussian benchmarks at 32K tokens, the FP8 kernel with head size 128 reaches 2,672 TFLOP/s versus 1,944 for AITER, equal to 53% of the peak stated in FP8 datasheets, while AITER stops at 39%. On the MLA shape the ratio rises to 1.99x (2,464 versus 1,237 TFLOP/s).
On energy efficiency, Zyphra reports 1.94 and 1.91 TFLOP/J at captured levels for Mistral-7B and Qwen3-8B, versus 1.60 and 1.56 for AITER. On precision, ZYFA's FP8 error is stated to be 0.22 times that of AITER on Mistral-7B and 0.38 on Qwen3-8B, with a KL divergence from the fp64 reference of 3.4×10⁻³ versus 5.9×10⁻³ in the full Mistral-7B model. The company also reports bit-for-bit identical outputs across 7,200 runs. All these values are claims by the authoring company and not results replicated by third parties.
Why FP8 alone is not enough
The technically most interesting part is the explanation of the bottleneck. FP8 matrix instructions run at twice the BF16 peak, but they accelerate only the two matrix products QKᵀ and PV; the intermediate softmax remains on the vector unit at the same speed in both precisions. With 32 queries versus 128 keys, the FP8 matrix work requires 1,024 cycles, while the exact softmax consumes 896: almost perfect overlap is therefore needed to keep the matrix units occupied. Zyphra's traces show that with AITER the matrix pipeline remains idle for 31.7% of cycles, also because the MI355X does not track matrix instruction completion at the hardware level and instruction order must create the overlap.
ZYFA intervenes with three choices: an approximate softmax called Lean P, which exploits the structure of e4m3 bits to compute 2^x with a multiply-add and an integer conversion, reducing vector cycles per block from 896 to 384; a two-wave ping-pong schedule that shares each SIMD between issuing matrix instructions and computing probabilities, raising matrix pipeline occupancy to 86% of cycles at steady state versus 60%; and per-block scaling of 64 keys via powers of 2, absorbed by CDNA4's scaled matrix instruction, with an additional matrix instruction for summing the rounded probabilities. Zyphra also notes that a Hadamard rotation on Q and K roughly halves the input error due to quantization, at a cost of about 0.1% of attention time on Mistral-7B.
Limitations and open questions
The picture has precise boundaries. The declared worst case is 1.16x on Mistral-7B, and at that level ZYFA's clock drops to 1,628 MHz versus 2,108 MHz for AITER: it depends on the number of bits that change, because AITER turns more small probabilities into zeros or subnormals, causing fewer bits to toggle. ZYFA compensates with fewer total cycles, but comparisons between different vendors come from different inputs and clocks and, above all, Zyphra did not measure NVIDIA GPUs. Portability beyond MI355X is not indicated, and adoption depends on independent verification of performance and error on other configurations.
