FuTransformer cuts Llama decode step to 2.8 ms on H100

FuTransformer ran complete Llama-3.1-8B decode at 2.8 ms per step at batch one on one Nvidia H100, in tests reported in an Oct. 16 arXiv preprint by Wei Wang.

The Oct. 16 preprint, not yet peer reviewed, reported 8.1 ms for the compiled transformers implementation, making FuTransformer about 2.9 times faster on that measure, according to Atlas360 calculations.

The work targets a central bottleneck in Llama-style inference. Prefill, the phase that processes prompt tokens, reuses weights across tokens and is dominated by matrix multiplication and activation data movement. Single-token decode has less reuse and depends more on weight and KV-cache reads.

FuTransformer is a CUDA implementation of a Transformer block for Nvidia Hopper GPUs. CUDA is Nvidia’s programming platform, and the KV cache stores attention keys and values used during generation.

The implementation uses FP8, an 8-bit floating-point format, for prefill and INT4, a 4-bit integer format, for decode projection weights and a KV-cache copy. It also folds fixed RMSNorm gain into weights, writes FP8 output directly with delayed scaling and supports prefill without padding.

On one H100, prefill reached 1.29 PFLOP/s and was up to 1.48 times faster than compiled PyTorch FP8. Its FP8 path was 2.02 to 2.18 times faster than the evaluated BF16 CODA path, while its BF16 path for gate/up projection with SwiGLU at the 8B shape was 1.13 times faster than CODA.

For decode, FuTransformer was 1.3 to 2.4 times faster than the faster PyTorch BF16 or INT4 block across 8B and 70B shapes at batches 1 to 32. That benchmark range spans a nearly ninefold difference between the 8B and 70B shapes, according to Atlas360 calculations.

The decode results also showed FuTransformer at 2.1 to 3.3 times faster than compiled, CUDA-graphed PyTorch BF16 and 1.3 to 6.3 times faster than the same block using PyTorch’s INT4 projection kernel. It was 31% to 51% faster than the FuTransformer FP8 path.

The implementation arranges weights and keys in Tensor Core fragment layouts, allowing direct dequantization into matrix-multiply operands without full intermediate tiles. A block-diagonal Hadamard transform is used to reduce KV-cache quantization error from outlier channels.

The speed gains came with an accuracy trade-off in the reported benchmark. Decoding every WikiText-2 token through the INT4 weight path raised perplexity, where lower is better, from 6.24 to 7.23 with round-to-nearest weights or to 6.99 with GPTQ in the same format; the rotated INT4 cache added 0.20.

The kernels matched a quantized reference computation within 0.001 perplexity. The reported results cover one H100, 8B and 70B shapes, and batch sizes from 1 to 32, leaving broader system-level performance outside the supplied figures.

Source: arXiv preprint, arXiv, Oct. 16, 2026