AI Engineering Productivity Benchmark™
Every engineer we deploy is benchmarked, and the code they ship is assessed: output, rework and quality scored from the code itself and compared with 120,000+ engineers worldwide. You see what you are paying for, not just hours. Based on research conducted at Stanford University.
use AI at workGoogle DORA, 2025
Experienced developers with AIMETR randomised trial, 2025
After AI, in one 355-engineer organisationStanford research, 2025
typical net gainonce the extra rework is subtracted · Stanford research, n = 2,562
- engineers in the benchmark data
- 120,000+
- companies across 10 industries
- 600+
- agreement with a panel of 10 senior reviewers
- r > 0.85
- of the engineers we deploy are benchmarked
- 100%
Why the usual metrics mislead
Traditional measures vary more than three times for the same work, and they reward the wrong behaviour.
- Lines of code
- More lines are not more output, and the count rewards verbose code.
- Commits and pull requests
- They reward splitting work into tiny pieces, not delivering more.
- Story points
- Subjective, easy to inflate and not comparable across teams.
- Self-assessment surveys
- A weak predictor of real output (r = 0.17). People misjudge their own productivity by about 30 percentile points.
- DORA metrics
- They measure delivery speed and DevOps maturity, not what is delivered.
Same work, four measures
Four equivalent pieces of work. Activity counts disagree by more than three times; output units rate them the same.
| Engineer 1 | Engineer 2 | Engineer 3 | Engineer 4 | |
|---|---|---|---|---|
| Lines of code | 2,029 | 583 | 1,511 | 802 |
| Commits | 21 | 12 | 6 | 26 |
| Story points | 5 | 13 | 3 | 6 |
| Output units | 10 | 10 | 10 | 10 |
A sample read-out, explored
Seven views from an anonymised read-out. Hover or tap the charts, switch the series, and move the sliders.
Talent mix by performance quartile
Bottom 25%Middle 50%Top 25%
- +13.6%Pull requestsMore PRs, not more output
- +1%Effective outputNo meaningful change
- 2.6×ReworkMost of the extra code
- −9%Code qualityVariance up 3.6 times
Output
Cost
Sample: an anonymised composite of real engagements and published research. Values are redrawn from the report charts and rounded.
AI adoption, maturity, spend and impact
Every vitX engineer works with AI agents, and independent studies find AI gains are small and uneven. The benchmark shows whether the agents add real output, or only more activity.
- 90%of tech professionals use AI at workGoogle DORA, 2025
- −19%experienced developers were slower with AI, while believing they were 20% fasterMETR randomised trial, 2025
- 15–20%typical net gain from AI coding tools once the extra rework is subtractedStanford research, n = 2,562
How we see AI in the code
No surveys. AI use is read from the Git history itself, then set against output, rework and quality.
Evidence of AI in each commit
Every commit is checked for signs that AI was involved. Pick a signal to see which commits it catches.
- claude/add-retry-testsAdd retry tests for payment webhooksAgent branch
- feature/search-filtersFaceted search on the product listConnected tool
- fix/login-timeoutIncrease the session timeout for SSO usersNo evidence
- codex/refactor-pricingSplit the pricing service into two modulesAgent branch
- hotfix/null-currencyGuard against a missing currency codeNo evidence
6 of 8 commits show AI evidence
“AI-assisted” means evidence that AI was involved, not a claim that AI wrote the code.
Adoption by team, quarter by quarter
Share of engineers with evidence of AI use. The rollout rarely spreads evenly.
| Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | |
|---|---|---|---|---|---|---|
| Team A | 5 | 15 | 35 | 60 | 75 | 85 |
| Team B | 0 | 5 | 10 | 20 | 40 | 55 |
| Team C | 10 | 30 | 45 | 55 | 70 | 90 |
0%90%
AI maturity: how well, not how much
Each repository is placed on a four-level ladder. Weighting by output shows whether the most mature practice sits on the work that matters.
- L1Ad hoc AI use45%20%
- L2Grounded prompting30%25%
- L3Agent-augmented development20%40%
- L4Agentic orchestration5%15%
Before and after, against a 12-month baseline
Index, pre-AI = 100.
- Effective outputhigher is better118
- Output per engineerhigher is better112
- Rework ratehigher is worse135
- Churn ratehigher is worse120
Pre-AI = 100
AI spend per output unit
From each tool’s own billing. Lower is better.
- Tool AUS$4.2
- Tool BUS$7.8
- Tool CUS$11.5
US$ per output unit
Illustrative figures from the vendor edition of the benchmark. Works from Git alone on day one. Connecting tools such as GitHub Copilot, Cursor, Claude Code, OpenAI Codex, Windsurf or GitLab Duo adds usage and spend.
How it works
- 01
Read-only on your repositories
Connected to the repositories your vitX engineers work in: GitHub, GitLab, Bitbucket or Azure DevOps. Nothing changes in how your team works.
- 02
Every commit scored
A model calibrated against a panel of 10 senior reviewers scores output, complexity and maintainability, in 15+ languages.
- 03
Compared with peers
Each engineer against peers of the same seniority, in your industry (10 sectors) and company size (4 bands).
- 04
Read-out, then action
You see the results. If an engineer misses the benchmark, we replace them under the Fit-or-Swap Guarantee™.
Your data stays yours
- No source code is stored: code is analysed in memory and deleted straight after.
- ISO 27001 and SOC 2 Type 2 certified infrastructure.
- You choose company, team or individual views, and can switch individual views off.
- Nothing is shared with anyone without your written approval.
Comes with every vitX engineer
- Measured from their first weeks on your team
- Output, rework and code quality in one read-out
- Compared with 120,000+ engineers worldwide
- Fit-or-Swap Guarantee™ if an engineer misses the benchmark