METR “Average 19% increase in working hours” depends on productivity expectations
Research shows that artificial intelligence (AI) coding AI does not increase productivity for all developers. It has been pointed out that it may have a negative impact on increased working hours for skilled developers.
The AI Safety Nonprofit Research Institute Metr said in a recent randomized controlled trial it found that AI-based coding tools such as Github Copilot and Cursor Pro drop by an average of 19%. The experiment involved 16 skilled developers who contributed to the open source project, and a total of 246 real-world tasks were included in the experiment.
Participants predicted that using AI would reduce working hours by an average of 24%. However, the actual measurement time was the opposite. On average, the group using AI took nearly 19% longer. “It contradicts the general belief that using AI tools will speed up,” Metr said of the results.
METR pointed to increased latency for rapid writes and responses, limitations in AI contextual understanding in complex, large codebases, and inconsistent AI-based coding assistance systems as reasons for delaying AI use. In fact, we knew that developers would spend more time writing results to write AI and reviewing reviews rather than coding. For experienced developers, it may have taken more time to change the code proposed by AI without the AI suggestion because they are familiar with the code they are using.
Other previous studies have shown that AI coding tools increase on average by 21% and up to 56%.
However, the researchers explained that it is “just an observation at the moment,” adding, “this means that not all AI coding tools will increase productivity.”
(Silicon Valley correspondent Wonho-seop)
