FlashRT-PAI speeds physical AI inference up to 12.66x

FLASHRT-PAI, an inference engine for physical AI models, ran 1.3 to 3.7 times faster than official compiled paths in tests, according to an arXiv preprint published Oct. 20.

The claim matters for robots and other action-oriented AI systems because the authors say real-time, one-decision-at-a-time inference is constrained by pipeline latency; reduced-precision tests showed 3.38 to 12.66 times gains over official eager implementations.

Liang Su, David Chen, Jason Li, Kevin Guo and Teresa Lv describe FLASHRT-PAI as a system built around model-hardware pipeline co-design. The paper’s title frames their argument: “Fast Kernels Are Not Fast Pipelines for Physical AI Inference.”

In the abstract, the authors said, “Latency is dominated not by individual kernel computation, but by system-level overhead.” They cited many short chip-task launches, movement of intermediate data between operations and repeated computation across consecutive decisions.

FLASHRT-PAI combines three mechanisms, according to the paper. Deep kernel fusion with launch overlap reduces gaps between operations; an execution contract records the pipeline’s stage graph, update rates and named buffers; and execution-state capsules let the system snapshot, restore and fork persistent state.

A kernel is a small computation sent to a processor, and a directed acyclic graph is an ordered map of computation stages. The paper’s central point is that optimizing those calls one by one can miss costs that appear across the whole inference loop.

The authors said general-purpose graph compilers fall short because they optimize individual calls without enough visibility into state carried between decisions. FLASHRT-PAI also supports acceleration of unmodified PyTorch models through kernel replacement governed by parity and latency checks, according to the preprint.

The evaluation covered six policies on RTX 5090 and four on Jetson AGX Thor, the authors said. In LIBERO-10, a four-bit pi0.5 configuration reached 94.04% success on Thor and 93.32% on RTX 5090, compared with 92.4% for the measured official FP16 reference.

Those figures do not establish broad performance across all physical AI models or hardware. The paper is an arXiv preprint and has not been peer reviewed, and the reported benchmarks are limited to the authors’ selected policies, devices and precision settings.

The paper says FLASHRT-PAI is open source, distributed through Hugging Face Kernel Hub, and that its opt-in pi0.5 kernels have been adopted into the LeRobot framework.

Source: preprint, arXiv cs.AI, Oct. 20