AI Coding Tools: Do They Actually Speed Up Development?

The Impact of AI Coding Tools on Developer Productivity

Recent years have witnessed a significant shift in software engineer workflows, driven by the emergence of AI-powered coding tools. Platforms like Cursor and GitHub Copilot are designed to boost efficiency by automating code writing, bug fixing, and change testing.

AI Models Powering the Tools

These tools leverage the capabilities of advanced AI models developed by leading organizations, including OpenAI, Google DeepMind, Anthropic, and xAI. Performance on various software engineering benchmarks has seen rapid improvement for these models in recent times.

New Study Challenges Productivity Claims

A new study released Thursday by the non-profit AI research group METR casts doubt on the extent to which current AI coding tools actually improve productivity for seasoned developers.

Study Methodology

METR employed a randomized controlled trial, recruiting 16 experienced developers actively involved in open-source projects. These developers were tasked with completing 246 genuine tasks within code repositories they routinely contribute to.

Approximately half of the tasks were designated as “AI-allowed,” permitting the use of cutting-edge AI coding tools such as Cursor Pro. The remaining tasks prohibited AI tool usage.

Developer Expectations vs. Reality

Prior to undertaking their assigned tasks, developers predicted that utilizing AI coding tools would reduce completion times by 24%. However, the study revealed a different outcome.

“Our findings indicate that allowing AI actually increases completion time by 19% – developers were demonstrably slower when employing AI tooling,” the researchers reported.

Experience with AI Tools

It’s important to note that only 56% of the participating developers had prior experience with Cursor, the primary AI tool used in the study. While a substantial majority (94%) had utilized web-based Large Language Models (LLMs) in their coding processes, this represented the first dedicated experience with Cursor for some.

Developers received training on Cursor’s functionality in preparation for the study.

Implications for Future Productivity

Despite the training, METR’s results raise concerns regarding the widely touted productivity benefits of AI coding tools anticipated in 2025. The study suggests developers should not automatically assume that AI-driven solutions – often referred to as “vibe coders” – will instantly accelerate their workflows.

Potential Reasons for Slowdown

METR researchers identified several potential factors contributing to the observed slowdown. Developers spent considerable time crafting prompts for the AI and awaiting responses, rather than actively coding.

Furthermore, AI tools often encounter difficulties within large and intricate codebases, which were characteristic of the test environment.

Cautious Conclusions and Ongoing Progress

The study’s authors emphasize that their findings should not be interpreted as a definitive condemnation of AI systems. They acknowledge that AI may still accelerate workflows for many or most software developers.

Other large-scale studies have, in fact, demonstrated that AI coding tools can enhance software engineer efficiency.

The authors also recognize the rapid pace of AI development and anticipate different results even within a short timeframe, such as three months. METR has observed significant improvements in AI coding tools’ ability to handle complex, long-term tasks in recent years.

Remaining Concerns

However, this research provides another reason for cautious optimism regarding the promised advantages of AI coding tools. Previous studies have highlighted the potential for these tools to introduce errors and, in certain instances, security vulnerabilities.