Blog / Article

How we fix bugs and incidents with agents, in 1 shot

Every Sentry error, bug or incident starts 1 agent session. It ends in a tested fix, a diagnosis or no action. Half our customer bugs close in 1 day.

This is the second post in our series about how we made Pensero agentic, in every part of the company. The first post was about customer operations. This post is about what comes after a customer finds a problem: the fix.

Before, a bug went through 4 or 5 people. Somebody saw the alert or the complaint. Somebody wrote the ticket. Somebody found the cause. Somebody wrote the fix. Somebody told the customer. Each hand-off added a day.

Today 1 agent session does the full path, in 1 shot: from the signal to a tested pull request, with the evidence. A person reviews it and merges it. Half of our customer bugs now close within 24 hours, against 1 in 4 before.

This is how we got there.

What "1 shot" means

1 signal starts 1 agent session. The session does not stop to ask a person for the next step. It gives back 1 of 3 results:

  1. A pull request with a test that fails before the fix and passes after it.
  2. A diagnosis with the cause, the evidence and the suggested fix, when the change is too large to make alone.
  3. "No action" with the reason, when the signal is noise.

A person sees the result only at the end. The person does not start from a blank page.

Step 1: Sentry errors

Every production error goes to Sentry, and Sentry posts it in our Slack channel for errors. An agent reads it in the thread:

  • Is it real? The agent counts the events and the users hit, and checks the region and the environment. An error with 1 event in 35 million requests gets no fix. A sandbox error is not a production error.
  • What is the cause? The agent reads the stack trace, the code on main and the recent changes. It finds the line, and the pull request that added it.
  • What is the smallest fix at the source? The agent writes the test that shows the error, then the fix. It does not add a try/except to hide the error.

An example from 19 November. Sentry issue PENSERO-DJANGO-18B showed a chart task that stopped at its 120-second limit, 85 times in 7 days. At 16:40 an agent started on it. It found that 79 of the 85 events stopped in the same calculation: a function called a slow replay of the full history, and did not use the result. At 16:46 the pull request was open. The function now reads the stored value, and a test proves that the values do not change. The review bots checked it, and a person merged it at 18:12.

In the last 60 days, 125 error sessions started from Slack. 61% of them ended with a merged pull request. The median session took 1.4 hours and cost less than 3 dollars.

Step 2: Bugs that customers report

In the first post, our supervisor agent sorts every escalated conversation into 5 classes. Class C is a bug. For a bug, the same agent continues:

  1. It collects the evidence: the steps, the screenshots from the conversation, and the code paths.
  2. It opens the Linear issue with a "Suggested fix" section: the likely cause, the files and the change.
  3. It opens the pull request itself when the fix is small and certain: a label, a copy change, or a few lines that a focused test proves.
  4. It routes the issue. A routing agent reads the issue and assigns the engineer who owns that part of the product.
  5. It closes the loop. When the issue is done, an agent writes the reply for the customer. A person sends it.

When a data problem has many old rows, the fix is not a script that a person runs one time. The fix goes into the daily checks, so the next scheduled run heals all the rows of that class, for every customer.

Step 3: Incidents

An incident is not 1 error. It is many signals at the same time, in many tools. So we gave the agent a battle plan.

The battle plan collects the signals of a time window from Slack alerts, Sentry, New Relic, Linear, Intercom and our admin health checks. It removes the noise, groups the signals by topic, and sorts the topics by customer impact. For each topic it proposes the action.

The root cause analysis takes 1 topic and gives: a summary that a non-engineer understands, the cause, the expected evolution, the earlier occurrences, the customers hit and the evidence. After a person approves it, the agent writes the Linear issue and 1 sub-issue for each action. Each sub-issue then goes through step 1 or step 2.

All incident work is read-only by default. The agent looks. It does not change production.

What makes 1 shot possible

The model is not the difference. The context is.

  • The rules live in the repository. Our coding rules, our writing rules and our review gate are files that every agent reads. One rule changes every agent the same day.
  • Skills, not prompts. Each job, such as Sentry triage, root cause analysis or the Help Center update, is a skill in the repository, with its own scripts. Python calculates the numbers. The agent writes the words.
  • Test first. A bug fix starts with a test that reproduces the bug. No test, no fix.
  • Fix the class, not the case. The agent changes the shared function once. It does not add a patch for 1 caller.
  • The review gate is automatic. 2 review bots and the CI check every pull request before a person sees it. The agent fixes their findings itself.
  • Every pull request says what else must happen: the Help Center change, and the manual steps after deploy in every environment.

1 rule

An agent never merges. The agent investigates, tests and proposes. A person reads the pull request and merges it. The agents do the work. People own what goes to production.

The results

Customer bugs (Linear issues with the "Customer Issues" label), before and after the intake agent started on 20 October:

July to September 20 Oct to 19 Nov
Customer bugs closed within 24 hours 25% 49%
Customer bugs closed within 7 days 75% 88%
Median time to close a customer bug 89 h 25 h

Sentry errors, last 60 days:

Error sessions started from Slack 125
Sessions that ended in a merged fix 61%
Median session time 1.4 h
Median cost of a session < 3 USD
  • Customer bugs close 3.5 times faster: the median went from 89 hours to 25 hours.
  • Half of the customer bugs close on the same day.
  • Agents opened 9 in 10 of our last 1,000 merged pull requests.

What we learned

  • Remove the hand-offs, not the people. The time was in the queue between people, not in the fix.
  • Ask for a result, not for help. A session that ends with a tested pull request, a diagnosis or "no action" is easy to review. A session that ends with a question is not.
  • Put the context in the repository. Agents are as good as the rules and the skills that they read.
  • Say no to noise. An agent that fixes every error makes noise. The count of events and users decides first.
  • Keep the merge human. Review is fast when the evidence is in the pull request.

Bug fixing is one part of the company. The next posts in this series show the other parts. It is the same transition that we help our customers make: agents do more of the work, people do the work that needs judgment, and you measure the result to know that it works.