LLM crisis reports capture 5.4%-11.0% of official figures

Sachin Kumar reported in an arXiv preprint submitted Oct. 13 that LLM-written crisis situation reports captured 5.4%-11.0% of same-day official figures across three primary disasters.

That is roughly one in 19 to one in nine official figures, according to Atlas360 calculations, in a study comparing 121 human reports with 261 generated reports across five disasters and three hazard types.

Kumar wrote, "Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare." The paper used LLM-assisted, author-approved qualitative content analysis to build a 15-slot schema from professional humanitarian situation reports.

Situation reports, often called sitreps, are structured crisis updates used to record conditions, needs, response activity and authority. The study assessed human-written and machine-written versions against the schema, rather than relying only on text-overlap measures.

On Cyclone Idai, google/gemini-3.7-flash judged the strongest machine arm at 0.827, compared with an empirical human ceiling of 0.821. The paper said the gap was not statistically reliable, with a 95% bootstrap confidence interval from -0.049 to 0.059 and a permutation p-value of 0.852.

That ranking also did not repeat under openai/gpt-5.6-luna, according to the preprint. On a blind 100-item Idai sample, the author matched the Gemini judge at 0.910 accuracy and unweighted Cohen's κ of 0.799, a chance-corrected agreement statistic.

All four judges met the plan's 0.7 raw-agreement threshold, Kumar reported. Only Gemini reached 0.7 on chance-corrected κ, limiting how broadly the judging result can be read from the supplied abstract.

The grounding results were weaker than the completeness scores suggested. For the three primary events, raw exact-number matching linked 14.3%-19.6% of official figures and 85.4%-90.1% of machine figures to the social stream.

A manual audit partly supported that pattern, the paper said. It confirmed 45 of 50 declared absences, 4 of 25 official matches and 24 of 25 machine matches semantically, showing a large difference between literal number overlap and source grounding.

The paper also found a provenance shortfall in all 12 observed event-judge cells. Report date, organisation, series and authority did not reach professional-report coverage in any cell, according to the arXiv abstract.

Common evaluation metrics performed poorly in the study's framing. For the three primary events, ROUGE-L was uninformative, BERTScore was negatively associated with completeness after correction for multiple comparisons, and a generic LLM judge changed sign.

The arXiv version is a preprint, not a peer-reviewed journal article. The listing says the work was accepted at the LT4CPR Workshop @ AACL-IJCNLP 2026, and the supplied material gives no deployment results or dated follow-up release.

Source: arXiv preprint, arXiv, Oct. 13, 2026