The use of artificial intelligence tools to assist with writing code is creating a messier and less sustainable product compared to human-authored code, according to a new report.

Generative artificial intelligence tools such as ChatGPT are increasingly being used to assist in writing code.

GitHub launched the Copilot tool to the public in 2022 which automatically suggests code as a user writes and is built on OpenAI’s code-specific language model Codex. Copilot is trained using publicly-available code, and works through an extension installed into the editor or development environment.

A new study by GitClear CTO Matthew Kloster and Alloy.dev Research CEO William Harding has aimed to investigate the impact of the use of AI on code development, and found potential issues around churn and tendency not to reuse existing code.

For the study, the researchers analysed 153 million lines of code from January 2020 to December 2023 and found “disconcerting trends for maintainability”.

According to GitHub, Copilot has led to 55 per cent faster code, 46 per cent more code being written and could add $US1.5 trillion to global GDPs, while junior developers are 20 per cent more likely to use Copilot than experienced developers.

This may imply that code produced with the help of Copilot is less sustainable, the report said.

“Experienced developers have the most informed understanding of how costly code will be to maintain over time,” the report said.

“If they are more averse to using AI suggestions, it raises questions about the extra code that junior developers are now contributing, faster than ever.”

A quick solution

AI tools offer coders a quick and easy solution, but this may be damaging in the longer term, according to the study.

“When developers are inundated with quick and easy suggestions that will work in the short term, it becomes a constant temptation to add more lines of code without really checking whether an existing system could be refined for reuse,” it said.

“To the extent that inexperienced developers continue to be offered implicit copy/paste suggestions via the tab key, the fix for this suggestion won’t be easy.”

A spokesperson for GitHub told Information Age it would be “misleading” to conclude from the report that AI tools are having a negative impact on code quality.

“There’s no way of knowing whether the changed lines of code GitClear analysed were authored by developers using AI tools, and the study does not account for any other factors that impact code quality,” the spokesperson said.

“Myriad factors impact code quality, and GitHub welcomes third-party research to explore the impact of AI tools for software development, like GitHub Copilot.”

Signs code may be AI generated

According to the report, the code analysis revealed three “red flags”: a decrease in the amount of code being moved around, an increase in copy/pasted code, and a significant increase in code churn.

Churn refers to the percentage of code pushed to the repository then reverted, removed or updated within a fortnight. The analysis found a near-40 per cent increase in code churn in 2023 compared to the previous year.

But churn is “relatively infrequent” when developers are writing their own code, with just three to four per cent of code churned before 2023.

“The more churn becomes commonplace, the greater risk of mistakes being deployed to production,” the report said.

“If the current pattern continues into 2024, more than seven per cent of all code changes will be reverted within two weeks, double the rate of 2021.”

There was also a 17 per cent year-on-year decrease in moved code, a statistic that commonly underpins code reuse. There is “little room to doubt the current implementation of AI assistants discourages code-reuse”, with the tools offering a “one-keystroke temptation to repeat existing code”.

“The benefits of code reuse are familiar to experienced developers – compared with newly added code, reused code has already been tested and proven stable in production,” the report said.

“Often, reused code has been touched by multiple developers, so is more likely to include documentation.”

There was also a more than 11 per cent increase in code being copy and pasted, something which the report authors said may be a “future headache”.