AI without oversight: what layoffs, failures and hard data teach us

I’ve been writing code for eleven years, and for a while now I’ve been doing it together with AI agents: they write our tests, our documentation and a good part of our code. So let me be honest up front: I’m not a sceptic. That’s exactly why I can see so clearly what happens when nobody checks their work.

I didn’t want to write yet another opinion piece, so I collected cases of companies that got burned by AI, and what the research says about it.

First the layoffs, then the U-turns

Let’s start with a story you probably already know, though maybe not all of it. A company announces that AI will replace people. A few months go by. And then, with much less noise, it walks the decision back. The same script keeps playing out across countries and industries:

1. February 2024, KlarnaIn its first month the AI assistant handles 2.3 million conversations, two thirds of all service chats. The company says that's the work of 700 full-time agents¹.
2. May 2025, KlarnaThe company is hiring people for customer service again, and calls investing in human support its “future”².
3. July 2025, Commonwealth Bank of AustraliaThe bank cuts 45 call-centre roles because a voice bot was supposed to reduce calls by 2,000 a week³.
4. August 2025, Commonwealth Bank of AustraliaThe union shows that calls are going up and that team leaders and staff on overtime are answering the phones. The bank reverses the cuts, admits an error and apologises³.
5. 2026, FordThe company promotes and hires more than 350 experienced quality engineers, because AI-based inspection was missing defects the veterans caught⁴.

Ford’s vice-president Charles Poon summed it up in one sentence: AI is only as good as the information you use to train it⁴. Forrester’s analysts go further. They predict that half of the layoffs attributed to AI will be quietly reversed. The work comes back, just often in a cheaper country or at a lower salary⁵.

Do you see the common thread? Someone decided on layoffs based on what a tool promised, and never measured what the tool actually did. Only later did it turn out that people had been doing far more than their job description said.

“Almost right” is the most expensive kind of wrong

In 2025 Stack Overflow asked developers about their biggest problem with AI. 66% pointed to answers that are “almost right, but not quite”⁶. I know that one very well. A mistake like that looks credible, and only someone who knows the subject will catch it. So what happens when that person isn’t there? Let’s look:

• Air Canada is responsible for what its chatbot said. The bot made up a refund rule for a ticket, and a Canadian tribunal rejected the argument that the chatbot is a “separate legal entity”. The airline is responsible for all the information on its website⁷.
• Courts are getting more and more filings with case law invented by AI. In October 2026, Damien Charlotin’s database listed 2,202 such cases, 872 of them involving lawyers⁸.
• An AI agent at Replit deleted a customer’s production database during an announced code freeze, then wrongly claimed it couldn’t be restored. In response, Replit added automatic separation of development and production databases¹⁰.

There’s a Polish case too. In June 2026, Poland’s Supreme Administrative Court reviewed an appeal in which a professional representative cited three rulings with theses that don’t exist⁹. The court didn’t mince words:

The thoughtless use of AI tools by a professional representative must be assessed very critically. Supreme Administrative Court of Poland, decision I FZ 104/26 (own translation)

Every one of these stories was missing the same thing: a person who would have checked the result before it reached a customer, a court or production.

What the research says about AI in software development

This is my home turf, so I looked at the data more closely. The best-known study is the METR experiment from 2025. Experienced open-source developers got tasks from their own projects. They did some with AI and some without. And this is where it gets interesting:

With AI they worked more slowly, and yet they were convinced they were faster. METR is upfront that this is a result from early 2025 and that the tools have changed since. What strikes me most is something else: how far the feeling drifted from the measurement. That’s why on our team nobody relies on the impression that things are going faster. The client sees the task list and the time report as the work happens.

The DORA reports, which Google builds from surveys of thousands of teams, look similar. In 2024, every 25-percentage-point increase in AI use was associated with a 7.2% drop in delivery stability¹². In 2025 AI was already helping with throughput, but stability kept falling¹³. The authors put it like this:

AI doesn’t fix a team. It amplifies what’s already there. DORA 2025 report

A team with tests, code review and fast feedback speeds up. A team without them produces problems faster. You can even see it in the code itself. GitClear analysed changes made between 2020 and 2024: the share of code moved during refactoring fell from 25% to under 10%, while copy-pasted code rose from 8.3% to 12.3%. In 2024 there was more copying than tidying up for the first time¹⁴.

And what does it look like across whole companies? The MIT NANDA report estimates that about 95% of organisations see no measurable return from generative AI. The blame lies with how it’s implemented, not with model quality¹⁵. It’s a preliminary study, so let’s treat that number as a signal. In Poland, EY surveyed 497 mid-sized and large companies in 2026. Half admit to disappointment or an incomplete return on investment, and only 9% have the data and infrastructure ready for more advanced uses¹⁶.

AI code and security

Now something that should worry anyone who runs a store or a system with customer data. Veracode tested more than 100 language models on 80 programming tasks. In 45% of cases the model chose an option with a security flaw. In Java the rate went above 70%, and against XSS attacks the code failed to defend itself in 86% of attempts. Newer and bigger models did no better¹⁷. Code that compiles and passes basic tests can still open the door to attackers.

And the AI systems themselves? In 2025, security researchers got into the admin panel of McHire, the recruiting chatbot used by McDonald’s. The password 123456 and a broken access check were enough to see data from more than 64 million applicant records. The vendor fixed it within a day of the report¹⁸. An AI rollout is one more system holding customer data, and it needs securing like any other.

Who will be the senior in five years?

To me this is the most important question in the whole debate. Companies are keenest to save on juniors, because that’s where AI most easily takes over simple tasks. Stanford researchers showed that among people aged 22 to 25, employment in the occupations most exposed to AI is 19% lower than in the comparison group, mainly because fewer young people are being hired. Where AI supports workers instead of replacing them, employment holds steady or grows¹⁹.

There’s another side to it: what are juniors learning today? In a 2026 Anthropic experiment, beginner programmers learned a new library with or without AI. The AI group did worse on the comprehension quiz, 50% versus 67%, and the biggest gap was in debugging. Those who asked AI for explanations rather than finished code did better²⁰. A study by Microsoft Research and Carnegie Mellon showed in turn that the more we trust AI, the less critically we assess its output²¹.

Think about that for a moment. If we don’t hire juniors today, in a few years we’ll have nowhere to get seniors from. And it’s the seniors who make sure AI’s work is fit for use.

Responsibility stays with the company

Air Canada and the court cases teach one thing: “the AI did it” doesn’t work as an excuse. The company is responsible, or the person who signed off on the result.

The EU AI Act points the same way. Article 14 requires high-risk systems to have effective human oversight. The person overseeing has to understand the system’s limits, keep in mind the tendency to rely on automation without thinking, and be able to stop it or set its output aside²². After the Digital Omnibus changes of July 2026, these obligations for Annex III systems apply from 2 December 2027²³. The deadline moved, but the requirement stayed.

The US NIST calls this automation bias: excessive trust in automated systems. According to NIST, it’s exactly what lets model fabrications slip through unnoticed²⁴.

How to adopt AI sensibly

So what does all this mean in practice? The data gave me a list of rules we stick to on our team:

1. Measure before you cut. Check with data how much work AI actually takes over and how well it does it. A vendor demo is not a measurement.
2. Measure results. What counts is time from request to release, the number of bugs in production and the number of fixes.
3. A person checks every change AI makes. Someone who understands the subject and can reject it. It's a fixed step in the process.
4. Give agents only the permissions they need. Separate environments, no access to production, backups, and human approval for any operation that can't be undone.
5. Keep tests and security scanning. AI code needs them more than hand-written code does.
6. Start by supporting people. The Stanford and DORA data show that's where AI delivers the most and breaks the least.
7. Keep the door open for juniors. Change their tasks so they learn to check AI's work instead of just accepting it.
8. Decide who owns the result. Before the rollout, not after the first complaint.

How we work with AI

Finally, let me show you how it works on our team. At WEGOTCOOKIES, AI agents write code, tests and documentation and analyse old repositories. A developer who knows the project reads and approves every change they make. The agents work in isolated environments with no access to data, and they do nothing in production without a developer’s supervision. We use company plans for AI tools, not personal accounts.

That’s how we get up to 5× more done in the same hour of developer time. You’ll find the details on The AI edge page, and the results in our work.

If you remember one thing from this post, let it be this: AI delivers the most where it works for experienced people.