Blog / Other resources

What did our AI budget buy?

Seats, tokens and agent runs are on the invoice. Measure what each one delivered: cost per delivery point, with quality, for the board.

Why does the AI invoice not answer the question?

Seats, tokens, agent sessions and contractor hours all land on the invoice. Each one counts what you used, not what you got. A team with high usage can deliver less than a team with low usage.

What does the research say?

  • Self-reports are not reliable. In a 2025 randomized trial by METR, experienced developers believed that AI made them 20% faster. The measurement showed that they took 19% longer. The METR update of February 2026 says that speedups now seem likely, but that study design can no longer measure them reliably. Surveys alone cannot answer the board.
  • Count completed work. Field experiments at Microsoft, Accenture and a Fortune 100 company, with 4,867 developers, measured completed tasks, not feelings. Completed tasks rose about 26% with an AI coding assistant.
  • Check quality with output. A 2025 study of 807 GitHub projects that adopted Cursor found a large but short increase in velocity, and a lasting increase in code complexity and static analysis warnings.

Why is hours saved a weak number?

Hours saved come from surveys or estimates. Freed time is not delivered work: it only counts if the team uses it to ship. The board cannot check the number.

Why is cost per merged PR not enough?

A merged pull request can be a one-line fix or a change across many systems. If you divide spend by the number of pull requests, a team that splits its work into small pull requests looks cheaper. Score each pull request by size and difficulty first.

Step 1: Collect all spend in one place

Collect AI tool seats, model and token spend, agent runs, and the cost of people and vendors. Use the same period for all of them.

Step 2: Score the delivered work

Score every finished pull request and ticket in delivery points: magnitude × complexity. The score is the same for a person, an AI tool and an agent. Pensero reads AI use from tool telemetry, not surveys.

Step 3: Read cost per delivery point, with quality

Divide spend by points for each team, tool, agent and vendor. Read rework and defects by source next to it. Cheap points that come back are not cheap.

Step 4: Put the next dollar where it returns most

Scale what delivers at a lower cost per point. Move other work to a lower-cost model or to an agent. Stop what does not deliver. Measure again each month.

What should the board see?

One page: total AI spend, delivery points, cost per delivery point, the trend since the last quarter, and rework by source. Add the mix of work by AI level, from L1 human-only to L5 fully autonomous agents, to show where the organization stands.

How Pensero measures AI spend

Sources

Frequently asked questions

How do you measure the ROI of AI in engineering?

Compare all AI spend with the finished work that it delivered, in one unit, with quality next to it.

What is cost per delivery point?

Total spend divided by the delivery points delivered in the same period. Read rework and defects next to it.

Is cost per merged PR a good AI metric?

Only in part. A merged pull request can be a one-line fix or a change across many systems, so score each pull request by size and difficulty first.

Are developer surveys enough to measure AI impact?

No. Surveys measure how people feel. In a 2025 trial, developers believed AI made them faster when the measurement showed they were slower. Use telemetry and delivered work.

Can you compare AI tools, agents and people?

Yes, if one platform scores all finished work the same way. Then compare cost per delivery point and quality by type of work.

How do you know that quality held?

Read rework and defects by source next to the delivery points. If they rise with the output, part of the output is not finished work.

Does this work for vendors and contractors?

Yes. Pensero scores their work in the same unit as the work of your own teams.

Can Pensero run in our own environment?

Yes. Pensero runs in Pensero Cloud, an EU deployment, your own cloud account or on-premises.