Define the result to improve
Choose one or two primary outcomes per pilot and define quality guardrails. Trying to improve every metric at once makes the result difficult to interpret.
- Reduce time from confirmed requirements to the first reviewable result.
- Reduce code-review waiting and fault-location time.
- Reduce rework caused by missed boundary conditions.
- Improve feedback time from automated tests and builds.
- Improve delivery frequency without increasing high-severity defects.
- Reduce drift between deployment documents, interface specifications and code.
Practical metrics and formulas
- Task lead time = time the task reaches its agreed done state minus time it becomes executable. Record waiting for requirements, data or interfaces separately.
- Flow efficiency = active work time / task lead time × 100%. Use it to locate waiting, not to score individual busyness.
- First-review pass rate = changes without major first-review rework / comparable changes × 100%. Define major rework before measurement.
- Defect reopen rate = defects reopened because the original issue remains / closed defects × 100%, excluding requirement changes.
- Change failure and deployment rework: record emergency fixes, rollbacks and unplanned deployments, separated by production, integration and acceptance environments.
- Engineering completeness: check tests, interface documentation, deployment, migration, rollback and change records against a risk-appropriate checklist.
Use metrics in groups
- Pair generated code volume with review rework, defects and maintainability to avoid rewarding unnecessary code.
- Pair first-submission speed with first-pass rate and tests so validation is not shifted downstream.
- Pair merge count with completed business outcomes and change failures so work is not split artificially.
- Pair test count with fault injection, defect discovery and stability so duplicate or weak assertions do not look productive.
- Pair tool usage with cycle time, quality and team evidence; activity is not benefit.
Design a comparable pilot
- Select similar tasks by system, stack, complexity and risk; do not compare a copy edit with a critical-service refactor.
- Fix definitions for start, finish, pause, rework and failure, and record tool version, configuration, participants and environment changes.
- Measure the baseline first, then introduce one primary change. Simultaneous tool, process, staffing and architecture changes cannot be attributed reliably.
- Collect both quantitative data and review or retrospective notes to explain task fit, rejected suggestions and added validation burden.
- Set stop conditions. Pause expansion when severe defects, data risk, rework or review burden crosses the agreed threshold.
What a monthly evaluation table should contain
Mark insufficient samples explicitly instead of presenting unstable percentages. Before cross-team comparison, confirm that application type, release method and risk level are comparable.
- Median lead time for comparable tasks, with task count and start/end definitions.
- Median review waiting time, with reviewer availability and queue context.
- First-review pass rate, with the major-rework definition.
- Defect reopen rate, with requirement changes excluded.
- Change failures or emergency rework, including severity and handling time.
- Documentation and deliverable completeness, including checklist version.
Relate investment to verified benefit
Record tool licenses, model usage, integration, training, review and governance cost, then compare them with verified time changes and reduced rework. Use consistent labor definitions. Time saved is not automatically realized financial value unless capacity is actually reassigned or cost is reduced.
Why do lines of code fail as a productivity measure?
Lines of code do not represent requirement value, complexity, quality or maintenance cost. More code may be duplicate or unnecessary and can increase later review and testing work.
Can a team use an industry benchmark directly?
Industry research can explain metrics but should not replace the team's own baseline. Application types, stacks, release cadence, risk and organization differ, so direct comparison can mislead.
How long does it take to determine whether a pilot works?
It depends on task volume and delivery cycle. The test is whether enough comparable tasks have reached review, testing and follow-up defect observation, not a universal number of days.
Should AI usage rate be an individual performance target?
Do not use prompt count, token use or login frequency directly as an individual target. It can encourage unnecessary use and hide quality, collaboration and delivery outcomes. Evaluate team process results and controlled experiments instead.
Sources and scope
- DORA software delivery performance metrics: https://dora.dev/guides/dora-metrics/
- DORA 2025 research on AI-assisted software development: https://dora.dev/research/2025/dora-report/
- NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
- This article provides a measurement method, not ASWORK or client productivity results. Conclusions require reviewable baselines, sample scope and definitions.