ColaBench tests AI agents across 349 personal-memory tasks

Wenjie Zhang and six co-authors introduced ColaBench, a synthetic AI benchmark with 349 tasks and 13,033 documents, in an arXiv preprint published Oct. 23.

The work targets personal agents, software systems meant to use a user's prior interactions when answering new requests, across 10 profiles in health and daily living.

The authors said current benchmarks have advanced memory recall, personalized retrieval and memory-guided action separately, while joint evaluation of knowledge retrieval and user-specific application remains less developed.

“Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances,” the authors wrote in the preprint.

ColaBench uses an Evidence Graph, a structure that links personal conditions to earlier disclosures, records which documents the user previously encountered and connects answer requirements to both personal evidence and newly acquired document evidence.

The authors also introduced MemKB, a method that uses state summaries to guide queries and rule application, and memory-document links to retrieve knowledge tied to the user's history. MemKB reached 86.25% accuracy with an average of 4.07 interaction rounds per task, according to the preprint.

The reported gains were larger on tasks that required combining information across multiple historical facts, the authors said. Their error analysis identified unsupported personal assumptions and incorrect rule application as barriers to using retrieved evidence.

The preprint has not been peer reviewed.

Source: arXiv preprint, arXiv cs.AI, Oct. 23