Self-Evolving Agent Framework Tackles the Consistency Gap on AppWorld

The self-evolving agent framework detailed on arXiv starts from a brutal measurement called the consistency gap.

On AppWorld with a ReAct agent using GPT-4.1, average per-run success is 77% but only 53% of tasks succeed in all five runs.

That 24-point shortfall is not noise, it is the deployment risk.

The authors, Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru and Malgorzata Zimon, do not propose a new model.

They propose memory.

At the core are two pieces: a Consistency Analyzer that pinpoints unstable, low-consistency steps likely to flip across executions, and a Guideline Generator that converts that diagnosis into targeted guidelines.

Those guidelines are committed to episodic memory and injected into future agent executions on similar tasks.

The diagnosis is familiar in production, where failures often occur not because the spec was wrong but because the agent was not given clear instructions, constraints or behavior definitions and then cannot generalize consistently across inputs.

On same-task evaluation, the framework raises the fraction of tasks that succeed in all five runs by 16 points.

On similar-task generalization, the gain is 13 points.

No claim is made about lifting single-run accuracy.

The claim is narrower and more operational: fewer flips when you run the same job again.

That is the right metric if you want an agent you can trust without a human in the loop.

The evidence is also narrow: AppWorld only, ReAct only, GPT-4.1 only, five runs per task.

It does not show whether the analyzer transfers to other benchmarks, other scaffolds, or models tuned for determinism.

It also does not price the overhead of generating, storing and retrieving guidelines at scale.

Even with those limits, turning specific unstable steps into reusable guidelines beats vague self-reflection.

Measure the all-five rate alongside the average, and you will see where your agent actually stands.