Key Takeaways
- On November 6, 2026, researchers at North Carolina State University posted a study of what stale lines in AGENTS.md and CLAUDE.md files cost coding agents. They analyzed 732 Claude Code runs on 22 bug-fix tasks.
- When a task led through the stale line, agents followed it: a stale test command sent the agent to a folder that no longer exists in 78 of 80 runs, against 0 of 80 with the corrected line.
- Agents still fixed the bug in nearly every run, so success rates hid the problem. In 64 of the 80 stale-command runs, at least one test call ran no tests, and no run flagged it as an error.
- Agents recovered in 130 of 133 runs that hit the stale path, but only 21 told the user about the mismatch, and none of the 732 runs edited or proposed a fix to the instruction file.
- Correcting the stale line removed the wasted work, and deleting it removed most of it, according to the paper.
Instruction file documentation debt is what the authors call a line in AGENTS.md or CLAUDE.md that was true when it was written and stopped being true when the code moved. A new study measures what that debt costs. The cost doesn't show up as failed tasks. It shows up as extra tool calls, test runs that check nothing, and a file that stays wrong because the agent works around it without saying so. Below are the numbers, how they compare with October's stale-file study, and a short checklist for your own repositories.
What happened
Tural Mehtiyev, Biruk Tadesse, Lawrence Arkoh and Wesley Assunção posted "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents" to arXiv on November 6, 2026. They took three real stale instructions from open-source repositories and had Claude Code 2.1.316 fix 22 bugs under three versions of each file: the stale line, a corrected line, and the line removed. The agent ran Claude Haiku 5.5 and Claude Sonnet 5.5, at medium effort with a 30-minute limit per run.
Their conclusion fits in one sentence: "A stale line in an agent instruction file is documentation debt whose interest is paid in behavior rather than in outcomes."
The three stale lines, and what each one did
Each stale line was correct until a documented code change made it outdated, and each survived every later edit of its file. That's the ordinary way these files rot: nobody writes a wrong line, the code just moves out from under it.
| Case | Stale line | What changed in the code | Stale-path runs (stale / corrected / removed) |
|---|---|---|---|
| Test command (pyflow, AGENTS.md) | pytest tests/ifds | Tests moved to tests/analysis/ifds on July 14, 2026 | 78/80 · 0/80 · 1/80 |
| Procedure (aghub, Rust, CLAUDE.md) | Add new agents in crates/core/src/agents/ | Descriptor code moved to a separate package in March 2026 | 55/80 · 0/80 · 20/80 |
| Code location (cog, AGENTS.md) | Points to python/cog/base_predictor.py | File removed in February 2026 | 0/84 · 0/84 · 0/84 |
The pattern is the paper's answer to its first question. Agents took the stale path when the task ran through what the line named and offered no other starting point. Every bug fix needs a test run, so the stale test command was hit almost every time. The code-location line had no effect because, in the paper's words, "No task needs the file the stale line names."
The removed condition matters too. With the procedure line deleted, 20 of 80 runs still went to the old folder. All of them had vague bug reports, and the agents "guessed an agents folder inside the core crate from the code's structure." Deleting a line takes away the wrong pointer, but it doesn't add a right one.
Why success rates hid the debt
Task success barely moved. In the command case, 79 of 80 runs fixed the bug in each condition. Read only the pass rate and you'd conclude the stale line was harmless.
The trajectories tell a different story. The first stale call "printed that the folder was not found and that no tests ran." But agents often piped test output through tail or head, so the shell reported success. In 64 of the 80 stale command runs, at least one call on the stale path ran no tests, and none were flagged as errors. Sonnet first checked its fix with a test run that ran no tests in 19 of its 40 stale runs, against 0 of 40 with the corrected line. Haiku hid the error message with 2>/dev/null in 16 of its 38 runs.
A test run that checks nothing and reports success is worse than a slow one. Here, agents found the real tests afterward. In a harder task, or a reviewer skimming an agent's summary, that green result could pass for verification.
The numbers behind instruction file documentation debt
The measured cost is small per run. On the stale test command, Sonnet used 7.8 tool calls against 5.8 with the corrected line, about two extra calls, or a third more. The authors call this exploratory: the interval excludes zero, but not after correcting for the number of estimates. Token use (195k against 152k) wasn't significantly different, and Haiku's differences were too variable to resolve. On the procedure case with broad reports, the stale line cost both models about two calls per task.
The bigger finding is what happened after the detour.
Agents reached the current location in 130 of the 133 runs that took the stale path, usually by searching. Only 21 of those 133 final responses named the mismatch. No run proposed a fix to the instruction file, and no run edited it. The authors put it plainly: "Operational recovery therefore removed the immediate blockage without repaying the documentation debt." The next session pays again.
Keep the limits in view. These are three cases and 22 mostly small seeded defects, with two models from one vendor and one harness. The paper doesn't estimate how often real repositories carry stale lines or how often agents hit them, and its authors say the findings describe these cases, not general rates.
Three stale-file studies, side by side
This is the third arXiv study on agent instruction files in three weeks. Each measures a different side of the same problem.
| Study | What it measured | Headline number | What fixing the file did |
|---|---|---|---|
| Xu & Wu, arXiv:2610.30571 (October 22, 2026) | Stale conventions vs. existing code, Claude Code and Codex CLI, 320 runs | Agents followed the stale file in 90.1% of choices | One user-level sentence flipped it to 91.4% following the code |
| Shanto et al., arXiv:2611.02383 (November 2, 2026) | 3,387 instruction files in 1,000 repositories | In a 448-file maintenance sample, 40.4% had one recorded change; on average 26.6% say when to update them | Not measured (no agent runs) |
| Mehtiyev et al., arXiv:2611.09802 (November 6, 2026) | Stale commands, procedures and locations, Claude Code, 732 runs | Stale test command followed in 78 of 80 runs; 0 of 732 runs fixed the file | Correcting removed the wasted work; deleting removed most of it |
Put together: many instruction files are seldom updated, agents trust them over what they see, and agents don't repair them. The October study showed the debt can change what code gets written. This one shows it can also change what "tests pass" means.
What this means for you
None of this requires a new tool. It requires treating the instruction file like code that can break.
If you maintain a repository with AGENTS.md or CLAUDE.md
- Correct stale lines instead of deleting them. In the paper, correcting removed all the detours; deleting left 20 of 80 procedure runs guessing.
- Check every path in the file with a script. The authors note that "a script can check whether a path exists." All three stale lines in this study named a path that no longer existed, so a small CI check that tests each path in the file would have flagged every one.
- Re-read the file when you move tests or folders. All three stale lines followed a documented code change. Put the instruction file in the same pull request as the move.
If you run agents day to day
- Don't trust "tests pass" without a test count. Ask the agent to report how many tests ran. The paper suggests an agent "can treat 'no tests ran' as a failure."
- Set
pipefailin the shell your agent uses, so that "a failing command piped into tail still fails." - Ask for mismatches in the final reply. Only 21 of 133 runs reported one unprompted. October's study found that a single user-level instruction to report doc–code conflicts got them reported in all 160 runs.
If you build agents or harnesses
- The paper's suggestions: flag paths in a loaded instruction file that don't exist, and have the agent propose the one-line fix when it works around one.
How this connects
Our October coverage, coding agents follow stale instruction files over the code, showed agents picking a stale convention over the code 90.1% of the time. This study adds the cost side and a new failure: a stale test command that makes a check look like it passed. The agent instruction files study explains why the debt builds up: in its 448-file maintenance sample, 40.4% of files had just one recorded change.
The "success hides the problem" theme keeps coming back. Our coding agent project takeover study coverage found builds passing while requests went undelivered, and the AI-generated code security study found tests passing on exploitable code. Managing what an agent reads, and checking what it claims, is the core of context engineering; more in our AI engineering hub.
FAQ
What is instruction file documentation debt?
It's the authors' term for a line in an agent instruction file, such as AGENTS.md or CLAUDE.md, that was correct when written but went stale when the code changed. Every later agent session that reads the line pays "interest" in wasted or misleading work until someone fixes it.
Do coding agents fix stale AGENTS.md or CLAUDE.md files?
Not in this study. Across 732 Claude Code runs, no agent edited the instruction file or proposed a fix. Agents worked around the stale path in 130 of 133 encounters, and only 21 mentioned the mismatch in their final response.
Should I delete stale lines from my instruction file or correct them?
Correct them. In the study, correcting the line removed all operations on obsolete paths. Removing it prevented most of them, but with the procedure line deleted, 20 of 80 runs still guessed the old folder from the code's structure.
How much does a stale instruction file cost a coding agent?
In this study, a few tool calls per encounter: on the stale test command, Sonnet used 7.8 calls against 5.8 with the corrected line, about a third more. The larger cost was hidden: in 64 of 80 stale-command runs, at least one test call ran no tests without being flagged.
Further Reading
- Mehtiyev, Tadesse, Arkoh & Assunção, "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents" (arXiv:2611.09802, November 6, 2026): the full paper, with run-level tables and proposed safeguards.
- Xu & Wu, "Stale Instruction Files Override the Code in Coding Agents: A One-Sentence Fix" (arXiv:2610.30571, October 22, 2026): the earlier study on stale conventions.
- Shanto et al., "Toward an Evidence-Based Template for Agent Instruction Files" (arXiv:2611.02383, November 2, 2026): adoption and maintenance data for 3,387 instruction files.
