Productivity and patience: How GitHub Copilot is expanding development horizons
GitHub, the shared repository of choice for developers of all types, has reshaped the playing field for AI-based development with its Copilot offering. The organization estimates that many developers employ the tool to generate between 30% and 50% of the code they eventually put into production.
Recently, researchers at Scalefocus launched a four-month experiment, involving three teams, to measure the worthiness and viability of GitHub Copilot. Their conclusion was Copilot "accelerates the development process, reduces the time spent on routine coding tasks, and enhances productivity. On average, its introduction boosts productivity by up to 20%."
Also: From AI trainers to ethicists: AI may obsolete some jobs but generate new ones
However, the researchers added, "limitations were also observed; Copilot occasionally struggled with more complex problems, requiring humans to guide it toward the desired solution."
Understanding Copilot's mechanics and best practices "is essential for maximizing its potential," they cautioned. "Recognizing that Copilot may not be equally relevant across all tasks is key to avoiding inefficiencies. Identifying areas where Copilot excels can significantly enhance its value, recognizing where it falls short will save a lot of wasted effort."
Copilot shines in "implementing straightforward, well-defined components in terms of performance and other non-functional aspects. Its efficiency diminishes when addressing complex bugs or tasks requiring deep domain expertise."
Also: The US awards 12 regional 'Tech Hubs', aiming to fund America's next Silicon Valleys
GitHub itself puts the developer productivity gain at about 55%, Mario Rodriguez, senior vice president of product at GitHub, said in a recent interview hosted by Sam Ransbotham of Boston College and Shervin Khodabandeh of Boston Consulting Group, published in MIT Sloan Management Review.
But productivity isn't the only metric that drives GitHub's intentions with Copilot. "We even try to ask a lot of our customers to really measure developer happiness overall," Rodriguez said. "If developers are unhappy in an organization, not a lot of things are happening, right? You want to keep them happy. Talent out there is scarce, so you definitely want to keep your developers happy."
The key to unlocking happiness is that Copilot is designed to be responsive and empathetic. "Copilot is very patient," he explained. "It doesn't get upset if you ask it 'You know, how does it actually work to do X, Y, and Z within this code base?' or 'I forgot how to write this register. Can you just please tell me what's the best way of doing that?'"
As a result, Rodriguez continued, "People that use it end up being significantly more productive. You see the difference at times in the code quality, and then you see the difference as well in their happiness because they don't have to write 10,000 lines of tests when Copilot can help them with that."
Also: Is AI in software engineering reaching an 'Oppenheimer moment'? Here's what you need to know
Copilot's greatest challenge is context, he pointed out. "Code and code development has a lot to do with the context that you're dealing with. Are you in a legacy code base or not? Are you in COBOL or in C++ or in JavaScript or TypeScript? It's a lot of context that needs to happen for the quality of that code to be high and for you to accept it."
GitHub Copilot, the first Copilot to emerge as an AI-powered development assistant, is incorporated into a division called GitHub Next. Rodriguez said this focus on AI is an entirely different proposition for the GitHub team. Typical GitHub code is vetted, tested, and deterministic: "We know if something is working or not. We know if there's a bug or not. We know how to fix, many times, that bug. We could see even the error that we're making while developing the feature."
Large language models, on the other hand, "are not deterministic," he continued. "And you cannot just expect them to be deterministic. You have to rethink how you're thinking about quality. You have to rethink how you're approaching the feature." At GitHub, "we have all of these evaluations and offline evaluations that we do for GitHub Copilot. And that's how we determine the quality of what it is giving you in something like code completion."
Also: AI is transforming organizations everywhere. How these 6 companies are leading the way
Ultimately, on some level, everybody will have the opportunity to be a developer, Rodriguez said. "The definition of a software developer will start changing going forward," he said. "Probably all of us know how to cook, or all of us know how to bike, or all of us know how to swim, but, look, I am not opening a Michelin-star restaurant tomorrow."
The impact on software development from AI will be subtler: "What if a text box is all they needed to be able to accomplish something that creates software and something that they could then derive value from?" For example, said Rodriguez: "If I could say very quickly in my phone, 'Hey, I am thinking of talking to my daughter about these things. Can you give me the last three X, Y, and Z articles and then just create a little program that we could play as a game?' You could envision Copilot being able to help you with that in the future."
Taking things further, "imagine, then, you could share this little program that Copilot created for you. And you could remix it, and you keep remixing and remixing. Artists do this all the time, by the way. I think tools like Copilot are going to help you do that. It's going to be unbelievable what our children are going to experience through the advance of this technology."