LLM repair actions gain from paired fixes in 76 of 114 traffic tests

Hengyu Ye and Songhua Hu proposed a counterfactual test for LLM-generated repair actions in an arXiv preprint published Oct. 20, with paired fixes outperforming single-part corrections in 76 of 114 traffic actions.

The Oct. 20 arXiv preprint matters for AI systems that suggest operational corrections: recombined parts recovered 61.6% of a reference action's loss reduction, 3.6 percentage points more than choosing only between original proposals.

The 61.6% result means the recombined approach captured a little under two-thirds of the benchmark improvement on held-out observations. The 76 traffic cases equal about two-thirds of the 114 tested actions, according to Atlas360 calculations.

The authors define an LLM-generated repair action as a recommendation that "specifies which records to change and by how much." In their framing, a poor overall repair can still contain one useful component: the right target records, the right numerical adjustment, or neither.

The paper evaluates that split in traffic networks and databases, where broad measurements can hide which underlying records are wrong. The authors compare combinations from the same starting point against a reference action, meaning a benchmark correction for the affected records.

In the traffic tests, correcting both the target set and the numerical update produced a combined gain larger than the sum of correcting each part separately in 76 of 114 actions. In six of those actions, either single correction made performance worse, while the two-part correction improved it.

The database results point to a different failure mode. Replacing only the model's proposed update with the reference update worsened 17 of 108 actions, and the authors said all 17 involved selecting unaffected records.

The work addresses a practical evaluation problem for AI repair tools: judging only the final outcome can miss whether a failed suggestion has reusable pieces. The proposed approach separates the target selection from the update amount, then tests how those components behave when recombined.

The limits are material. The paper is an arXiv preprint and has not been peer reviewed.

The authors also tested whether components from two model proposals could be reused together. Their held-out result, 61.6% versus 58.0%, suggests a modest improvement from recombination within the reported setup, not proof that LLM-generated repairs are reliable in live systems.

Source: arXiv cs.AI preprint, Oct. 20, 2026