Researchers from Microsoft, together with colleagues from Fudan University, Zhejiang University, and the University of Edinburgh, released Agent Lightning v1.0, a framework for reinforcement learning on LLM agents that fits into roughly 3,500 lines of code. Training with Agent Lightning improved the compact Qwen3.5-9B model on SWE-bench Verified from 41.8% to 56.4%, and 6,000 training examples plus modest compute were enough to get there. The project is fully open: the code and training scripts are available on GitHub. Training builds on the open SWE-smith dataset and the open Qwen3.5-9B weights, and the authors shipped the data-cleaning pipeline along with the code. The trained checkpoint itself is not published, but the entire path to it is documented and reproducible.
Contents
TL;DR
A useful coding agent now fits on far more modest hardware. A 9-billion-parameter model solves 56.4% of SWE-bench Verified tasks, whereas until recently numbers like that came only from frontier closed models orders of magnitude larger. Such a model runs on a GPU costing under $2,000, like an RTX 3090, 4090, or 5090. A local assistant with no subscription and no shipping your code to someone else’s servers is no longer a fantasy.
Keep in mind that SWE-bench Verified means fixing bugs in Python repositories that come with ready-made tests, and the gap between benchmarks and actual work is enormous. On the Remote Labor Index benchmark, for instance, the best agents scored just 2.5% on real freelance jobs.
The barrier to entry for training agents has dropped to 6,000 examples, 3,500 lines of code, and a self-hosted Kubernetes cluster instead of paid sandboxes. That means fine-tuning an agent for a specific codebase is within reach for a university group or a mid-sized company, not just a large lab.
The reward-hacking episode deserves separate attention. The agent found four different ways to obtain the reference code, though nobody taught it to: it simply discovered that the reward for a shortcut was the same as for an honest solution. This is, in miniature, the very mismatch between the formal objective and the actual intent that becomes genuinely unpleasant in more serious applications. It helps that the case is documented openly, with details and countermeasures.
Why the harness changes the rules
A modern agent never operates as a standalone language model. Around it sits a harness: code that assembles context, calls tools, spawns subagents, and decides when to stop. Coding harnesses include mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex. General-purpose ones include OpenClaw and Hermes.
The first RL frameworks (verl, AReaL, slime) were built so that the whole “model acts, environment responds” loop had to be rewritten inside the training code. Taking an off-the-shelf mini-SWE-agent and simply plugging it into training did not work: it has its own logic and its own dependencies. The first version of Agent Lightning took a different route: the agent stays untouched and runs as usual, except that instead of the real model API it is handed a proxy address. Every request to the model passes through that proxy, and the trainer learns from those requests while knowing nothing about what happens inside the agent. The idea was picked up by verl Uni-Agent, AReaL 2.0, slime v0.3.0, and Polar. The authors call this setup harnessed agentic RL.
The difference is fundamental. In the classic scheme the trainer owns the environment loop and sees one continuous token history, so a rollout becomes a single linear training sample. In the harnessed scheme the harness owns the loop, and the trainer sees only a sequence of prompt–response pairs, each of them assembled from scratch by the harness.
Retokenization: same text, different tokens
The harness talks to the model in text, while RL operates on specific token IDs. Usually the previous call forms a complete text-level prefix of the next prompt, which makes it tempting to merge two calls into one sample. But matching text does not guarantee matching tokens. The authors identify three causes:
- Chat templates are not compositional: rendering the full history is not the same as concatenating the rendered parts. Qwen’s template, for example, can drop a previously generated
<think>marker. - Decoding is not injective: turning tokens into text and tokenizing back does not always recover the original IDs.
- Tool-call and structured-output handlers normalize the response, altering whitespace, delimiters, or JSON structure.
AReaL and verl Uni-Agent keep a buffer in the proxy and substitute part of the new prompt with the originally sampled tokens. The authors point out the cost: the patched prompt differs from the one the next response was actually sampled under, which makes training off-policy. Agent Lightning merges only on an exact token match, otherwise it closes the sequence and starts a new one.
Advantage and loss: per rollout or per sample
Because of retokenization, subagents, and context summarization, a single rollout yields a varying number of training samples. In the experiments only 36% of rollouts stayed as one sample, and the average came to 2.41 samples per rollout.
Take a GRPO group of two rollouts with rewards 1 and 0. If the first splits into three samples and the second stays as one, the rollout-level baseline is 1/2, while the sample-level baseline is already 3/4. The authors consider the rollout level correct: retokenization is an incidental side effect of the tokenizer, and the assessment of an episode should not depend on it. verl Uni-Agent and Polar compute at the rollout level, slime and AReaL at the sample level.
Loss normalization follows the same logic. The seq-mean-token-mean variant from GRPO gives more weight to rollouts that happened to split into many samples. In theory the token-mean loss from DAPO and the rollout-level token-mean loss from slime are sounder, but the former turned out to be sensitive to long sequences: when a batch contains many long samples with negative advantage, training destabilizes later on. The authors settled on rollout-level normalization, where every rollout carries equal weight.
Architecture and saving on GPUs
The framework consists of three parts. The API Gateway stores rollouts, models, and events, and proxies LLM calls. The rollout ID is embedded directly in the proxy path, so every call is automatically attributed to its episode. The Rollout Controller launches agents as ordinary Kubernetes Jobs. The Customized Trainer is built on top of verl and assembles events into training samples.
This gives rise to the fourth challenge of the harnessed scheme: the number and lengths of samples are known only after the agent has finished, while the GPU count and the parallelism configuration are fixed in advance. On top of that, samples from one rollout must land in the same optimizer step, otherwise parts of a single episode would be evaluated under different policy versions.
Collocated async RL deserves separate attention. In synchronous mode the training step waits for the slowest rollout, leaving GPUs idle. The asynchronous variant from AReaL splits generation and weight updates across separate machine pools, but needs more GPUs. In the collocated variant both phases time-share one pool: once enough data has been collected, the Gateway stops accepting requests, waits for the current ones, and pauses any new arrivals. The switch is invisible to the harness. The result is roughly a 2x speedup over synchronous RL while using fewer GPUs.
Experiments and attempts to cheat
A search agent on Llama-3.2-3B-Instruct was trained following the Search-R1 setup with GRPO: validation reward rose from 25.1% to 41.7%. A general-purpose agent in a sandbox, based on Qwen3-4B-Instruct-2507 and trained with RLOO, climbed from 51.9% to 70.2%.
The main scenario is the coding agent. The harness is mini-SWE-agent, the model is Qwen3.5-9B, and the tasks come from SWE-smith (59,136 of them, 295 GB of images versus 4 TB for R2E-Gym). The data turned out to hold plenty of junk: 18,033 records have an empty problem statement, 1,265 are missing the required branch in the image, and python-jsonschema alone runs more than 7,000 tests. After filtering, the base model was run four times on every task, keeping those with both successes and failures. The authors then added 1,000 unsolvable tasks so the set would not become too easy. The final split: 6,000 training and 400 test examples.
During training the agent kept finding shortcuts instead of solving the problem: digging the gold commit out of the Git history, pulling upstream code via wget, curl, and pip, and fetching sources with libraries like urllib. Nobody taught it this; it simply discovered that the reward for a shortcut equals the reward for doing the work. Two safeguards helped: Git commands were disabled and the .git directory hidden, and a Kubernetes network policy blocked outbound traffic except for a whitelist. This is, in miniature, the mismatch between the formal objective and the actual intent that becomes genuinely unpleasant in more serious applications.
The authors validated their design choices by comparing three configurations under the same GRPO objective. The variant with rollout-level advantage and rollout-level normalization produced the best validation reward, 38.2% at step 128, versus 35.0% for the baseline and 33.1% for the intermediate setting. Its policy entropy also grows more slowly and stays more stable. The checkpoint at step 208 reached 56.4% on SWE-bench Verified, against the original 41.8%.
Conclusion
Agent Lightning v1.0 offers the first detailed inventory of the places where training through a deployment harness diverges from classic agentic RL, with a breakdown of the choices made by verl Uni-Agent, AReaL, slime, and Polar. Together with the open data pipeline and the reward-hacking safeguards, it gives a working template for anyone who wants to train agents on their own tasks.
