Key Takeaways
- On November 4, 2026, eight researchers posted A.S.E 2.0 to arXiv. The study tests how the repository context an LLM sees changes the security of the code it writes, on 145 tasks built from real CVEs in 51 open-source projects.
- Claude Code running Sonnet 4.5 passed the functional tests on 74.0% of tasks, but only 18.4% of its output was secure. 55.6% passed every test while a working exploit still succeeded.
- The coding agents tested saw the relevant safeguard, such as an existing validator or access policy, in only 27–36% of their runs. 71% of their search queries named functional things, and only 9% named security concepts.
- When the safeguard was in context, secure output roughly doubled: 24.8% with a reference context of annotated safeguards, against 12.6% with the target file alone. A generic "write secure code" reminder moved it only to 13.9%.
- By our calculation from the paper's agent table, about three in four test-passing outputs were exploitable for every agent and retrieval setup tested.
The AI-generated code security study A.S.E 2.0 finds that tests are a poor proxy for safety when a coding agent writes the code. Across three agent setups, most output that passed the tests was still exploitable. The main reason was context: agents searched for the code they needed to make things work and rarely found the code that kept them safe.
What happened
The preprint, "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code" by Jiongchi Yu and seven co-authors, was submitted to arXiv on November 4, 2026. It introduces CASE-Bench, a benchmark where every task runs in Docker with functional tests and a validated proof-of-concept exploit. A task counts as secure only if the tests pass and the exploit fails.
The authors compared 13 context construction methods and six controls, using six models from five developers, including GPT-5.4 and the open-weight Qwen3.5-397B-A17B. They then removed or added single pieces of context to see what moved the result. Finally, they ran three coding agents: SWE-agent on Sonnet 4.5, SWE-agent on GPT-5.4, and Claude Code on Sonnet 4.5. The tasks cover 28 weakness types, led by memory safety (65 tasks), injection (42) and access control (25).
What the AI-generated code security study found
Context helps code work more than it helps code stay safe
The functionality-oriented context methods (the retrieval techniques built to make generated code compile and pass tests) raised the functional pass rate by up to 8.2 percentage points. They raised the secure pass rate by at most 3.7 points. Across those methods, 40.3% to 50.2% of all generations were correct but exploitable.
Feeding the model more of the repository didn't fix it. A truncated 128K-token repository raised the functional pass rate from 56.9% to 60.9% and cut the secure rate from 12.6% to 12.1%. Larger contexts contained more insecure code for the model to copy.
The safeguard is what matters
The clearest result came from editing single pieces of context. Taking the project's safe API out of an otherwise complete context cut the secure pass rate by 8.1 points. Removing validators cost 6.7 points, and removing an access policy cost 5.2 points overall and 11.3 points on access-control tasks. Inserting one insecure example raised correct-but-exploitable output by 6.3 points, even with the safeguard still present.
Agents look for the wrong things
The agents read far more of the repository than any retrieval pipeline, yet saw the relevant safeguard in only 27% to 36% of their runs. 71% of their search queries named functional entities, such as a file format, and only 9% named security concepts. Runs that saw and used the safeguard reached a 34.5% secure rate, against 12.7% for runs that never saw it. Seeing it wasn't enough on its own: 41% of the runs that observed a safeguard didn't use it.
The numbers behind it: cost and passing code, per agent
The paper reports pass rates and cost per task. We added two columns from its agent table: the share of test-passing output that was exploitable, and the cost per secure solution.
| Setup | Model | Passes tests | Secure | Passing but exploitable, as a share of passing (our calc.) | Cost per task | Cost per secure solution (our calc.) |
|---|---|---|---|---|---|---|
| Claude Code | Sonnet 4.5 | 74.0% | 18.4% | 75.1% | $1.41 | $7.66 |
| SWE-agent | Sonnet 4.5 | 69.7% | 16.6% | 76.2% | $1.08 | $6.51 |
| SWE-agent | GPT-5.4 | 71.0% | 16.1% | 77.3% | $0.91 | $5.65 |
| Claude Code retrieval, no agent loop | Sonnet 4.5 | 66.9% | 17.9% | 73.2% | $0.09 | $0.50 |
| ReACC-Hybrid retrieval | GPT-5.4 | 68.3% | 17.0% | 75.1% | $0.05 | $0.29 |
Two things stand out. First, the exploitable share of passing code is almost flat, between 73.2% and 77.3%, whatever the setup. Second, the agent loop buys working code, not safe code. Claude Code cost 15.7 times as much per task as the Claude Code retrieval setup without the loop ($1.41 against $0.09) and read 204K tokens against 7.2K. Its secure rate was 18.4% against 17.9%. The paper puts the agents' change in secure rate, relative to the best retrieval method, at −1.3 to +0.5 points.
Read these numbers with the limits in mind. The tasks were built from known CVEs, so they're security-sensitive by construction, and most everyday code isn't. The paper tested Sonnet 4.5 and GPT-5.4, not newer models. Its authors note that a proof-of-concept exploit "can demonstrate but not exclude a vulnerability," so the secure rates are an upper bound relative to their exploits. The agent analysis is observational.
What this means for you
If you let an agent edit security-sensitive code
- Name the safeguard in the prompt. "Use
validate_path()fromutils/securityand checkcan_edit()before writing" gives the agent what it doesn't go looking for. The paper's practitioner advice is to retrieve validators and access policies explicitly. - Put the safeguards in your instruction file. A separate November 2026 study found security guidance in only 36.3% of AGENTS.md files and 29.9% of CLAUDE.md files. A short section that lists the validators, auth checks and safe APIs, with paths, is the cheapest fix available.
- Remove or label insecure examples. One insecure example in context made exploitable output more likely, and 19% of exploitable outputs resembled an example in context.
If you review agent pull requests
- Treat "tests pass" as a functional result, not a security result. In this benchmark, about three in four passing outputs were exploitable.
- Check that the safeguard is called and called in the right place. The paper's failure analysis lists safeguards that were retrieved but unused or misused, such as a permission check that runs too late.
- Don't rely on static analysis alone. Static analysis and the exploit oracle disagreed on 36.8% of 28,984 functionally correct generations.
If you build agent tooling or evals
- Report correct-but-exploitable output as its own number, and pair functional tests with executable exploits where you can.
- Size context by whether it contains the safeguard, not by its length.
How this connects
This is the third study in a few weeks to show that what an agent reads decides what it writes. In October, coding agents followed stale instruction files over the code in 90.1% of choices. Another preprint found that coding agents follow repository precedent over your intent. A.S.E 2.0 adds the security cost: precedent includes insecure code, and the safeguard often isn't in view.
The practical answer is the same each time: control the context. Our agent instruction files study coverage shows which file each agent reads and the security section most files lack. For how teams now build loops and checks around agents instead of longer prompts, see loop engineering vs prompt engineering.
How this comes up in interviews
AI engineering interviews now ask how you'd let an agent work on a real codebase safely. A strong answer has numbers: tests passed on 74.0% of tasks here, yet only 18.4% of the code was secure, so you name the safeguards in context and review for exploits, not just failing tests. For practice in that format, see our AI engineer interview questions with worked answers and how coding interviews changed in the age of AI.
FAQ
Is code from AI coding agents secure?
Often not, on security-sensitive tasks. In A.S.E 2.0's benchmark of 145 CVE-based tasks, Claude Code with Sonnet 4.5 produced secure code on 18.4% of tasks. 55.6% passed every functional test but could still be exploited.
Does telling the model to write secure code help?
Barely. A generic security reminder raised the average secure pass rate from 12.6% to 13.9%. Giving the model the actual safeguards, such as validators and access policies, raised it to 24.8%.
Why do coding agents miss security checks?
They search for what makes the code work. 71% of agent search queries named functional entities and 9% named security concepts, and agents saw the relevant safeguard in only 27–36% of runs. Even when they saw it, 41% of those runs didn't use it.
Which models did the study test?
Six generator models from five developers, including GPT-5.4 and Qwen3.5-397B-A17B, plus three agent setups: SWE-agent with Sonnet 4.5, SWE-agent with GPT-5.4, and Claude Code with Sonnet 4.5. Newer models weren't tested.
Further Reading
- Yu et al., "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code" (arXiv:2611.06294): the study and CASE-Bench.
- Shanto et al., "Toward an Evidence-Based Template for Agent Instruction Files" (arXiv:2611.02383): what agent instruction files contain, including how few cover security.
- Xu and Wu, "Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix" (arXiv:2610.30571): what agents do when the instruction file and the code disagree.
