A couple of weeks ago the US government put export controls on Fable Five, and just like that a frontier model that firms had started building on simply vanished. Anthropic couldn't really enforce who could and couldn't use it, so they withdrew it for everyone over a weekend. It had only been out in the wild for a fortnight, so I doubt many people had bet their whole stack on it yet. But it was a wake-up call, and I've been reflecting on what it actually exposed.
I had Abbas Khan from Togii on the podcast this week, and he framed the vulnerability better than I could:
"If we are a software business or a technology provider and we hitch ourselves to a certain model and suddenly that model just gets taken away, then what do we do?"
That's the thing. When you build in someone else's house, you are exposed to things way beyond your control. It doesn't have to be as dramatic as a government stepping in. The model might become too expensive, and the cost of tokens is a very live issue right now. It might stop being best in class. It might simply become unacceptable on security grounds. Any of those, and you are left holding a product that depends on a foundation you were only ever renting.
I think that is the beginning of a much bigger question. What does trust even mean now?
For most of my career, buying software has been built on trust, and that trust was guaranteed by a piece of paper and the legal jurisdiction you signed it in. You trust the vendor. You trust the certification, ISO 27001, SOC 2, Cyber Essentials. You stack up those layers until you can say, I trust this relationship and this vendor to deliver on their promise. Cloud went through exactly the same journey. Nobody trusted it at first, so organisations bought the on-prem version and took two years to migrate. Then the accreditations arrived, everyone got comfortable, and now every company is just cloud anyway.
There are a few distinct risks tangled up in all of this, and it helps to pull them apart.
The first is dependency. That is the Fable Five story. You build on a frontier model, it becomes central to your product, and then something well beyond your control moves the ground under you. It was Mr Trump that pulled the plug that time, and even Anthropic didn't feel able to say no.
The second is what Abbas calls intelligence sprawl. AI is being embedded into almost every application and workflow, often with different vendors running different models, and the effect creeps up on you:
"An organisation can gradually lose sight of where sensitive information is going, which systems are influencing decisions, and who ultimately controls the evidence behind those decisions."
The third is opacity. In a big organisation it is genuinely hard to see who took what, where they put it, what they asked and what they got back. You can sign a zero-training agreement with a frontier provider, and they will commit that nothing gets used for training, but you are taking them at their word. Sometimes you need something stronger than a promise. As Abbas put it:
"It's trust or verify. Sometimes you have to trust it because you have no option. But if you did have an option, and if you were able to verify it, that shows people they are in control."
And then there is the cost of not being able to see. When you can't trust the output, you have to check all of it, which steadily eats the efficiency you were chasing in the first place:
"You get a big verification tax. Yes, you got the answer to your question, it was lovely, but now you have to go and verify it."
A lot of these AI rollouts are being held up by confidence. The capability is rarely the problem. The tool does something amazing on dummy data, the proof of concept lands beautifully, and then it stalls the moment real client data needs to go anywhere near it. The block is trust.
That tells me something about what matters at this stage of the maturity curve. Explainability, transparency and visibility of what is going on are more valuable right now than raw capability. The intelligence has proved itself up to a point. The black box has not. Abbas described what firms respond to when they finally get that visibility, and I thought it was a lovely way to put it:
"Bringing a torch to that dark cave of throwing stuff into AI is something a lot of people are hungry for."
Getting trust back means giving people control of their own data, keeping the audit trail so everything can be forensically checked, and being able to prove where information went and where it did not. It also means being honest about where AI belongs and where it does not. Simon Ayley made this point about the water industry when he was on the podcast. AI is used all over the back office, but there is a wall between the AI and the water itself, because if something goes wrong with water, people drink it. That kind of separation of concerns is what a mature relationship with a technology looks like. It shows judgement.
And it means right-sizing the thing. You don't need the latest Opus or GPT model to summarise a transcript. A tiny local model on your own machine will do it, more cheaply and without your data leaving the building. Knowing which task goes to which model, and why, is going to matter more and more.
Capability is becoming table stakes. The models are extraordinary and they will only get better. The organisations that pull ahead will be the ones that can prove they are in control of it. How much can you verify. How much can you control. How much can you actually see. That is what will decide it.
I would love to get to a world where AI is trusted by default. For the foreseeable future though, we are going to have to be able to verify and control what goes in, and that is a good discipline to build now rather than after the first breach. Trust used to be a piece of paper. From here, it is going to have to be something you can see for yourself.