Why vendors are hard to compare today
- Each vendor reports its own numbers. Hours, story points and status slides use different units.
- Story points are not the same across teams. 10 points for one vendor can be 3 points for another.
- AI agents hide in the hours. A vendor can use agents for part of the work, and the invoice does not show it.
How to compare vendors in 4 steps
- Connect the code and the tickets. Read the data from the repositories and the issue tracker, not from vendor reports.
- Score the delivered work with one unit. Score each merged pull request and closed ticket by size and difficulty, the same way for every vendor.
- Compare cost and quality. For each vendor, divide the invoice by the delivered work, and read rework next to it.
- Compare by type of work. A vendor can be the best for new features and the most expensive for maintenance.
| What to compare | Vendor A | Vendor B | In-house team |
|---|---|---|---|
| Delivered work per month | Same unit for all | Same unit for all | Same unit for all |
| Cost per delivered unit | Invoice ÷ delivered work | Invoice ÷ delivered work | Salaries and tools ÷ delivered work |
| Rework | Share reverted or fixed again | Share reverted or fixed again | Share reverted or fixed again |
| Share of work by AI agents | Agent work ÷ all work | Agent work ÷ all work | Agent work ÷ all work |
How to measure how much work moved to AI agents
- Find who made each change. Each pull request comes from an engineer, an engineer with an AI assistant, or an agent.
- Score all of it with the same unit. Then the work of agents and people adds up on one scale.
- Divide agent work by all work. Track the share every month, for each vendor and each team.
See how to measure the cost and output of AI coding agents.
Common mistakes
- Comparing day rates. A cheap day rate can be the most expensive per delivered unit.
- Using each vendor's story points. They are not the same unit.
- Counting AI seats as AI work. A seat shows access, not delivered work.
How Pensero compares vendors
The Delivery Point is one unit of finished work, scored by size and difficulty, the same way for vendors, in-house teams and AI agents. Pensero reads it next to cost and quality, and traces it to the pull request and ticket behind it.
- See Pensero for service firms and the current pricing.
- Book a scoping session to compare your vendors on one scale.
Sources
Frequently asked questions
How do I compare the output of several software vendors?
Score the delivered work of each vendor with the same unit, from the code and the tickets, and compare cost per unit and rework.
Which tool measures what a software vendor delivered, not the hours it billed?
Pensero measures the delivered work of vendors, in-house teams and AI agents on one scale, the Delivery Point, with cost and quality for each.
How do I know how much of a vendor's work AI agents did?
Find who made each change, score all changes with the same unit, and divide the agent work by all work.
Can I compare vendors with my in-house team?
Yes, if the same unit scores both. Then you compare cost per delivered unit and quality by type of work.
