AGENTS.md considered expensive
Photo by Roger Starnes Sr on Unsplash
...providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average.
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (No.)
Effective marketing
Contexts files (AGENTS.md, CLAUDE.md) are included in over 60,000 open source repositories according to agents.md, as
well as every proprietary repository I've personally encountered. They are considered a best practice (by Anthropic
and OpenAI). They do not improve agents' effectiveness at completing tasks, despite being effective at directing agents'
behavior. Coding agents do follow the instructions in context files (sometimes), but the primary substantive impact of
this is increasing token usage, and thus cost. Or to put it another way, the primary substantive impact of context files
is a cash transfer from users to providers.
Handwritten files are "better"
However, developer-committed files outperform LLM-generated ones by a significant margin of 7% on average.
The handwritten (or edited) files still do not outperform nothing.
Against "best practices"
In the absence of any evidence of effectiveness, we should stop committing these files to version control. Deleting them might lead to a measurable positive impact on electricity and public water usage. And maybe the next iPhone won't cost up to $3200.