AI accelerates software development to breakneck speeds, but measuring that is tricky
Software development and deployment cycles keep accelerating, thanks in large part to artificial intelligence (AI), which can generate code and make suggestions. Even with such hyper-productivity, IT managers and business leaders remain perplexed about how to measure AI's impact.
That's the word from GitLab's new survey of 5,315 executives and IT professionals, covering software development productivity and DevSecOps. AI-assisted development is now the norm -- 78% of respondents say they are currently using AI in software development or plan to in the next two years, up from 64% in 2023, the survey found. In addition, 67% say their software development lifecycle is now mostly or completely automated.
Also: From AI trainers to ethicists: AI may obsolete some jobs but generate new ones
Bringing in AI may be accelerating software development toward blinding speeds. Stunningly, most executives (69%) indicate they are shipping software twice as fast as last year. Plus, it's actually taking longer to get IT professionals up to speed with the tasks at hand. More than half (52%) say it takes more than three months to onboard new developers -- up from 42% a year ago.
Upper-level executives are much more wary of AI than their staff members. A majority of executives (56%) say that introducing AI into the software development lifecycle is risky in terms of privacy and data security. In contrast, only 40% of professionals have such concerns.
Executives also worry more about AI skills, with 35% identifying a lack of appropriate skill sets to employ AI or interpret AI output as an obstacle to using AI. Only 26% of IT professionals agree.
Respondents currently using AI for software development (43%) were much more likely than those not using AI (20%) to say that developer onboarding typically takes less than a month. The survey found the same effect for DevSecOps platform usage, with 44% of those currently using a platform saying that developer onboarding takes less than a month, compared to 20% of those not using a platform.
Also: Meet your new IT superhero: Citizen developers flex their AI muscles
The survey also found that the most popular use for AI within IT shops is code generation, plus providing explanations on how code works. For future work, the largest number would like AI to help them achieve forecasting and productivity metrics.
How AI is used in development
- Code generation and code suggestion/completion: 47%
- Explanations of how a piece of code works: 40%
- Summaries of code changes: 38%
- Chatbots that allow users to ask questions in documentation using natural language: 35%
- Summaries of code reviews: 35%
What IT pros and managers want to see in AI
- Forecasting productivity metrics and identifying anomalies across the software development lifecycle: 38%
- Explanations of how a vulnerability can be exploited and how to remediate it: 37%
- Chatbots that allow users to ask questions in documentation using natural language: 36%
- Suggestions for who can review code change: 34%
- Fixing failed pipeline jobs: 31%
Software supply chain security is a potential weak spot, with 67% of professionals reporting a quarter or more of the code they work on is from open-source libraries. At the same time, only 21% of organizations currently use a software bill of materials (SBOM) to document the composition of their software.
Also: Apple coders, rejoice! Your programming tools just got a big, free AI boost
Executives say developer productivity is a critical operational metric, but many are unsure how to measure it. Slightly more than half of executives (51%) say their current methods for measuring developer productivity are flawed or want to measure it but aren't sure how. At least 45% admit they aren't even measuring developer productivity against business outcomes.
A majority of executives (55%) agree that developer productivity is important, and 57% agree that measuring this productivity is key to business growth. Only 42% currently measure developer productivity within their organization and are happy with their approach. More than a third (36%) believe their methods for measuring developer productivity are flawed, while 15% want to measure developer productivity but aren't sure how.