We tell our customers that the agentic transition is not about buying a tool. It is about changing how work is organized. So we did it to ourselves first, in every part of Pensero.
This is the first post in a series about how we did it. We start with customer operations, because there every mistake is visible to a customer.
Today 3 levels handle every conversation: an AI agent that answers, a supervisor agent that escalates and improves it, and people who own what is left. Our median time to resolution went from 16 hours to 6. Customers who reach a person get an answer 2.5 times faster than before. Satisfaction went up while volume grew more than 20 times.
This is how we got there.
Step 1: Fin answers first
We started with Intercom's Fin AI agent in front of every conversation. Fin reads our Help Center and answers in seconds, at any hour, in the customer's language. Today it resolves 85% of conversations on its own.
Customers loved it, and they used it. Conversations went from 0.3 a day to more than 5, and to 14 a day at the peak of our onboarding season. When the answer comes at once, people ask more.
Fin also showed us what a first-line agent alone cannot do:
- Nobody checked its work. When Fin gave a weak answer, we found out only if the customer complained.
- It could only repeat the Help Center. When an article was wrong or missing, Fin gave the wrong answer every time, until somebody noticed.
- Escalations got slower. Fin tried to hold on to the conversation before it handed it over. The conversations that reached a person took a median of 28 hours to resolve.
- It could not see the customer's data. Most of our questions are about numbers, and the Help Center does not have them.
A first-line agent is a good start. It is not a system.
Step 2: A supervisor agent that escalates and improves
Next we added a supervisor: a set of Devin automations that watch Fin's work.
It reviews every conversation. Every morning the supervisor reads all of yesterday's conversations and every low rating. For each one it decides: did the customer get the answer? If not, it finds the cause in our Help Center and in our code, and puts the conversation in 1 of 5 classes:
| Class | What went wrong | What the supervisor does |
|---|---|---|
| A. Fin guidance | The answer exists, Fin did not give it | Writes the guidance change for Fin |
| B. Help Center | An article is missing, wrong or unclear | Checks the facts in the code and fixes the article |
| C. Bug | The product misbehaves | Opens the issue with evidence and a suggested fix, and a PR when the fix is small |
| D. Configuration | Setup on our side or by the customer's admin | Opens the issue with what to configure |
| E. Feature request | The product cannot do it | Logs it as customer feedback for Product |
It escalates with the work already done. When a customer asks for a person, the supervisor classifies the problem, opens the ticket, links the customer and writes a draft answer. The person who picks it up starts from a diagnosis, not from a blank page.
It closes the loop. When the issue is fixed, the supervisor writes the reply that tells the customer.
It makes the first line better every day. A Help Center fix is live the same morning, so Fin gives the right answer to the next customer. A guidance change stops the same miss from happening again. In its first month the supervisor turned 72 unresolved conversations into 18 Fin guidance changes, 10 Help Center fixes, 16 bug reports, 18 configuration fixes and 9 feature requests. Every one is a gap that Fin no longer has.
Step 3: Customers check their own data with the Pensero MCP
Most support questions at Pensero are not "how does this work?". They are "why does my team show 42 delivery points last month?" or "is my repository connected?". The answer is in the customer's data.
Our support team does not have access to customer data, and we want to keep it that way. So we gave the answer to the customer, not to us.
We connected Fin to the Pensero MCP, the same server our customers already use from Claude, ChatGPT, Gemini or GitHub Copilot. When a customer asks about their numbers, Fin calls the MCP as that customer:
- Scoped to the person. Fin acts as the verified user. A manager sees their team, an engineer sees their own work, an executive sees the organization. The same role and scope rules as the product apply.
- Read-only. Fin can look. It cannot change anything.
- Scoped to the question. "My", "me" and "I" mean the person. A team name means that team. When the question is not clear, Fin asks one question with the options.
The customer gets the real number, for the real period, in the chat. No ticket, no screenshot, no wait.
3 levels, 1 rule
- Fin with the Pensero MCP answers from the Help Center and from the customer's own data.
- The supervisor agent reviews, classifies, escalates with a diagnosis, and improves level 1.
- People, our CX team and our engineers, handle what the agents cannot solve.
And 1 rule: an agent never sends a customer a message that a person did not approve. Fin answers in real time because it uses only approved content and the customer's own data. The supervisor writes only internal notes and drafts. A person reads the draft, changes it if necessary, and sends it. The agents do the investigation. People own the relationship.
The results
| People only | Fin only | 3 levels | |
|---|---|---|---|
| Conversations a day | 0.3 | 5.2 | 6.7 |
| Median time to resolution | 16.2 h | 7.5 h | 6.0 h |
| Time to resolve 90% of conversations | 43.9 h | 54.0 h | 28.2 h |
| Median time to resolve an escalated conversation | 17.1 h | 27.8 h | 11.0 h |
| CX Score (1 to 5) | 4.17 | 4.33 | |
| Conversations with a positive CX Score | 81% | 86% |
- Resolution time is down 63%, from 16.2 hours with people only to 6.0 hours.
- The slow tail is cut in half. 90% of conversations now close within 28 hours, against 54 hours with Fin alone.
- Escalations are 2.5 times faster. A person resolves an escalated conversation in a median of 11 hours, against 28 hours with Fin alone. The ticket, the diagnosis and the draft are ready when the person opens it.
- Satisfaction went up with volume. More than 20 times the conversations, and 86% of rated conversations are positive.
What we learned
- A first-line agent needs a supervisor. Measure the first line with your own review, not only with the vendor's resolution rate.
- Fix the source, not the answer. Every miss becomes a Help Center fix, a guidance change or a ticket. The same question should never fail twice.
- Give customers their own data, scoped to them. Most support questions are data questions. A scoped, read-only MCP answers them better than a person who must ask for access.
- Keep people where trust is made. Agents investigate and draft. People decide and send.
Customer operations is only one part of the company. We made the same change in every other part of Pensero, and the next posts in this series show how. It is the same transition we help our customers make: agents do more of the work, people do the work that needs judgment, and you measure the result to know that it works.