DualQuant may lower LLM memory use by 2.61x

Mahsa Salmani, Sayeh Sharify and Hesham Mostafa submitted an arXiv preprint on Oct. 9 describing DualQuant, a 4-bit LLM quantization method reporting up to 2.46x higher throughput versus BF16.

The Oct. 9 arXiv preprint said DualQuant also delivered up to 2.61x memory reduction in a packed W4A4 inference path for Llama-3.1-8B on an RTX 4090, compared with BF16, a 16-bit numerical format.

The work addresses a practical limit in low-bit inference for large language models. AI accelerators that can run 4-bit formats directly need both weights, the stored model parameters, and activations, the intermediate values produced during inference, to be quantized to capture the full speed benefit, the authors said.

The preprint’s central claim is that DualQuant can compress both parts without calibration data. Calibration data means representative inputs used before deployment to choose scaling values for quantization, and the authors said activation quantization typically depends on that step.

The authors wrote that DualQuant “matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats” in tests across Llama and Qwen models. They evaluated the method on WikiText-2, C4 and zero-shot downstream tasks, where a model is tested without task-specific examples.

DualQuant starts from weight-reconstruction mean squared error, a measure of the gap between original and quantized weights. The authors said the scaling factors that reduce that error also provide effective per-channel rescaling for activations before those activations are quantized.

The paper describes the optimization as non-differentiable and uses variable splitting with block coordinate descent, techniques that divide a problem into smaller update steps. The authors said each subproblem has a closed-form update, meaning the answer can be computed directly rather than through a long search.

The preprint said DualQuant supports several low-bit formats, including the full MX family and NVFP4. The authors said it is, to their knowledge, the first dual-scaling method with formal support for MX and NV block formats.

The hardware validation goes beyond simulated, or “fake,” quantization, according to the authors. They said a real packed W4A4 implementation for Llama-3.1-8B on an RTX 4090 closely matched fake-quantization accuracy while producing the reported memory and throughput improvements over BF16.

The paper was submitted as version 1 on Oct. 9 at 15:51:38 UTC and is listed by arXiv under Machine Learning and Artificial Intelligence. The arXiv page names Salmani as the submitting author and shows an arXiv-issued DOI pending registration.

The evidence remains preliminary. The paper is not yet peer reviewed, and the abstract reports the real packed W4A4 test on one named model, Llama-3.1-8B, and one named accelerator setup, an RTX 4090.

For AI infrastructure teams, the result points to a possible route for reducing inference memory needs and improving throughput on 4-bit-capable hardware without maintaining calibration datasets. The preprint does not show whether the same gains hold across larger models, other accelerators or production traffic patterns.

Source: arXiv preprint, arXiv, Oct. 9, 2026