In a world where competition is global, companies need to be more and more productive. Anything helps from agile practices, to more productive technologies, and lately, AI in the workplace promises huge productivity gains.

If you have been aware of what has been happening in the tech world in the last months, it seems that building software as we did until now is a dying trade.

Anyone with an idea can vibecode a world-class SaaS that will replace your CRM in hours just by hitting tab and writing requests and a lot of “don’t make any mistakes messages.

The reality is that this is just not true.

You can easily create nice PoCs that you can use to test your ideas, get the first customers or have a working prototype that non technical people can share with engineers so they can productify it.

But building software that can hold real business value is much more complex than that; playing Call of Duty doesn’t make you a Navy SEAL.

Tabs will make you free

As we discussed at my previous article LLM code assistants are great is you know what you are doing. They can amplify your code-writing capabilities to be more productive and effective, but they can’t do magic.

Even the best models have a significant percentage of wrong code: where the code doesn’t do what it’s supposed to make, and a non-negligible percentage of highly destructive code (can break all existing systems).

Let’s get some data, Dave

Since we have thousands of customers who are using Pensero to measure their tech operations, and as we incorporate both AI metrics and quality metrics, we were able to draw the correlation between AI usage and the number of bugs pushed to production that required further fixing.

This is a very simple way of getting an idea of what the net productivity gains from using AI are.

Here you can see two graphs, the first one is the usage of AI for a given set of users, and the second one is the percentage of code that required further fixing.

There’s a strong positive correlation between the amount of code that involved AI for its creation and the amount of bug that required fixing right after the release of that code.

We also observed that the correlation was much lower depending on the person’s expertise in the technology used.

We could see that non-software engineers were able to create software at a comparable rate to their technical peers, but also the percentage of bugs was several multiples higher.

No surprise here: experts are more productive and produce fewer bugs compared to non-experts, who may seem to be producing as much raw code but with a much higher rate of faulty code.

So then, is AI coding bad?

Absolutely not.

We observed that, when used properly, the net productivity gains were quite considerable and that teams using code assistants were more productive and required less time to get value in front of customers.

That being said, let’s consider the following: code assistants are just tools, and as with any tool, the quality of the output highly depends on the skills of the user.

There are also some pitfalls you need to be aware of.

Beware the Dunning-Kruger effect: LLM assistants can create a false sensation of safety and you may feel all the code you produce is greater than it actually is.

Coding is a critical task; a single faulty line can create mayhem, so make sure you check (and understand) every single line you create.

Until now it was recommendable to review the changes you were introducing in the pull request, as you are responsible to diff every file before committing them, now is even more critical, because the amount of non-requested changes introduced out of the blue is significant.

The same applies to testing, and here LLMs are great to create fixtures, mocks, tests, etc, but again, check them; 20% of them will be wrong or redundant.

Do not remove the human from the loop, just don’t. It’s cool to have an LLM reviewing your PR, but just get someone from the team to check it too, thank me later.

Measure all the relevant metrics, not just the amount of code shipped, but the quality of it, the incidents it generated, the impact on customer cycle, etc.

If you want to be more productive without creating more mess than value, we can help you with that. At Pensero, all of our customers get Bug and AI tracking correlated with their productivity out of the box.