Large language models can self-improve at web agent tasks

Led by Ajay Patel at the University of Pennsylvania, with Markus Hofmarcher, Claudiu Leoveanu-Condrei, Chris Callison-Burch and Sepp Hochreiter. I am the fourth of six authors. On arXiv as 2405.20309.

The problem

Prompting alone does not make a language model a competent agent. Long-horizon tasks that require a sequence of correct actions in a stateful environment defeat it, and fine-tuning is not an easy escape because the training data does not exist. Sequential decision-making traces are expensive to collect and expensive to label.

WebArena makes the difficulty concrete. An agent has to drive a browser over 812 tasks spanning shopping, forums, maps and a GitLab instance, and the strongest baseline in the original paper, GPT-4, solved roughly 14% of them.

A second problem sits underneath the first: even if you improve an agent, you cannot easily tell what changed. One aggregate completion rate cannot distinguish “the model can now do things it could not do before” from “the model got luckier at things it could already sometimes do.”

The idea

Self-improvement here means fine-tuning a model on data it generated itself, with the learning signal coming from unsupervised filtering rather than from labels.

The base model is Qwen-1.5-72B-Chat. Run it over all 812 WebArena tasks and keep the trajectories, then discard the ones that failed detectably: the model declaring a task impossible or that it cannot proceed, refusals in the form of stop[N/A] or an empty stop, environment errors, and generations that will not parse. What survives is called the set of plausible trajectories. No ground-truth labels are consulted anywhere in this step.

The filter is crude and it works. Of 812 trajectories, 58 survive, and the fraction that actually completed their task rises from about 7% to about 43%. Concentrating correctness sixfold using only signals the environment hands you for free is the mechanism the whole paper rests on.

From those trajectories two datasets are built. In-domain examples are individual steps, sampled to balance the beginning, middle and end of trajectories so the model does not only learn how to start. Out-of-domain examples are invented: the base model is asked to produce entirely new objectives on websites outside WebArena’s six, with hypothetical solution steps and synthetic page observations. A cosine-similarity ceiling stops the generated objectives collapsing into paraphrases of each other.

Three mixtures get fine-tuned with QLoRA: in-domain only, both, and out-of-domain only.

Why it matters, and how much

Both mixtures containing real trajectories improved. The best, the combined mixture, moved functional correctness from 7.14 to 9.36 — the 31% relative improvement in the abstract.

That number deserves its context, which the paper supplies. A trivial agent that immediately emits stop[N/A] and does nothing at all scores 4.68 on this benchmark, because some tasks are correctly answered by refusing. So the honest reading is that an unsupervised procedure moved a 72B open model about two absolute points on a benchmark where frontier models were then scoring in the low teens. Real, reproducible and small.

The out-of-domain-only mixture is the interesting failure. It lost ground on functional correctness, dropping to 6.16, produced trajectories roughly 1.6 times longer than the in-domain model’s and about 3.9 times as many invalid actions — yet it was the only variant to acquire certain capabilities no other variant did. Purely synthetic tasks taught the model things, and also taught it bad habits.

Iterating the procedure a second time did not help. The second round yields more data of lower quality, with a higher proportion of failed trajectories, and the twice-tuned model does not beat the once-tuned one. That matches other work on repeated training against self-generated data, and is worth stating plainly rather than leaving open.

Measuring what the score hides

The paper’s second contribution is two metrics, the part most likely to outlive the specific result.

The capability score exploits the fact that WebArena’s 812 tasks are instantiated from 241 templates, many of which are paraphrases. Grouping them by similarity gives 136 distinct capabilities, and a model scores a capability only if it solves at least one non-trivial instance of it. This separates breadth from robustness. The best mixtures gained five capabilities, a 24% relative improvement, smaller than the 31% gain in raw completion — which is itself informative, since it means part of the improvement was the model becoming more consistent at things it could already do. Capabilities were lost as well as gained; the net was positive for every mixture.

VERTEX-DTW scores trajectory quality independently of task completion. It extends the VERTEX score with a Dynamic Time Warping alignment step so trajectories of different lengths can be compared, plus a decay that discounts matches occurring far apart in the sequence. The honest caveat is in the paper: with no human reference trajectories available, GPT-4 output stands in as the reference, which caps how much the number can be trusted.

Where it applies

The recipe suits any environment that tells you cheaply when something went wrong but not when it went right. Compilers, test suites, API error responses and form validators all qualify. You need no reward model and no annotation, only a filter for detectable failure and enough attempts that the survivors are enriched for success. The corollaries transfer too: tasks the model invents for itself are riskier than its real traces, and a second round of the same trick is unlikely to pay.