SemiCat reaches 49.94% on TinyGSM in AI preprint

Oscar Davis and seven co-authors proposed SemiCat, a continuous diffusion language-model design that reached 49.94% accuracy on TinyGSM, in an arXiv preprint submitted Friday, Oct. 9.

The preprint matters because it targets a cost and performance trade-off in non-autoregressive text generation: DoubleCat improved TinyGSM accuracy more than threefold without an extra forward pass.

The authors frame continuous diffusion language models, or CDLMs, as an attempt to bring continuous flow modeling to language, with potential advantages including faster sampling and controllable generation. They say CDLM capabilities have started to rival discrete diffusion, while training choices remain less understood than in image or video diffusion.

The paper studies two choices: self-conditioning and embedding geometry. Self-conditioning means feeding a model its own intermediate prediction during training, a method the authors say reduces mismatch between training states and inference states.

The authors introduce DoubleCat, a two-level interpolant, to recover self-conditioning gains without adding another forward pass. In deep-learning systems, each forward pass adds compute, so eliminating one can matter if the method scales.

SemiCat adds discrete reveals to DoubleCat, directly addressing what the authors call token identifiability, where large-norm Gaussian noise can obscure which token a model should recognise. With standard training hyperparameters and a tuned sampler, SemiCat reached 49.94% on TinyGSM, according to the preprint.

At 1.7B scale, the authors say SemiCat beat discrete diffusion on four of six likelihood-based evaluations and outperformed one-hot flows in unconditional generation. One-hot flows represent tokens with sparse vectors; unconditional generation evaluates text produced without a prompt specified in the supplied abstract.

The embedding result is narrower but relevant for model design. The authors find cross-entropy, a common training loss, pushes embeddings toward a shape where decoding error closely tracks one-hot representations. That implies some benefits of one-hot geometry may emerge through training rather than requiring a fixed design.

The preprint also says the authors briefly examine hybrid corruption processes and noise schedules, including asynchronous schedules. Corruption processes define how data is noised during training; noise schedules set how that noising changes across model steps.

The main caveat is evidentiary. The arXiv manuscript is an early version and not yet peer reviewed, and the supplied material does not give independent replication, full benchmark tables, training cost, sampler details or production tests.

For AI companies and research labs, the result is relevant only if the cheaper design holds beyond TinyGSM and the evaluations named in the abstract. The commercial question is whether CDLMs can reduce inference cost or improve control without sacrificing the reliability associated with widely used language-model architectures.

The paper was submitted as arXiv:2610.13834v1 in computer science machine learning, with authors Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis Béthune, Pierre Ablin and Marco Cuturi.

Source: arXiv preprint, Oct. 9, 2026