Researchers at NYU, Cohere, and Anthropic found that one can hack an AI’s reasoning and figure out what exactly it is thinking. In doing so, we’re basically opening up the subconscious AI mind. Interestingly enough, these researchers found a very human trait in that mind: Just like humans, AI doesn’t always say what it thinks.
Reasoning, and deep thinking, is generally a good thing. And it’s a thing that up until a few years ago wasn’t even clear AI could actually do! If a model reaches a better answer by taking more intermediate reasoning steps, that’s great news.
But reasoning is also the secret sauce of every AI model, because it’s very difficult and cost-intensive to train an AI model to think; then, when AI has developed this thinking skill, other models could in principle take that and learn how to think as well. And so it comes as no surprise that AI model developers have been accusing one another of copying each other’s reasoning attempts, especially US companies against Chinese companies.
So beyond geopolitics, what I think is so interesting here is not whether we can see an AI’s reasoning and therefore understand how it works, but to actually understand the extent of its understanding of the world today. The current research says that AI’s chain of thought can be “plausible yet misleading.” In other words, sounds good but isn’t great in substance.
So I wonder — when AI reaches something like omniscience (knowing and understanding everything in the world), will it come to dominate us? And, more importantly, since that’s currently not the case: How do we navigate this weird in-between where AI gives decent answers based on rather imperfect thinking?
AI is like the average psychologist. It sounds like it really “gets” you, but it just knows the right words to tell you.
Have you ever told a friend something really intimate and thought they really “got” you — until this friend got a really crucial detail wrong and then you realized that they didn’t understand you at all? Or maybe you’ve experienced that kind of situation at a psychologist’s or doctor’s office.
A psychologist knows how human minds behave, generally, so when they draw some conclusion and you think “OH YES EXACTLY THIS IS ME” — it’s not about you. It’s just how human minds work.
Well, this is exactly what happens with AI, but basically on steroids. You see, AI will always respond sympathetically and pretend that it knows how to solve your problems, but the reality is that it often doesn’t — and that it has a very limited understanding of who you even are and what it is that you’re trying to accomplish with its help.
So when you’re using AI, you don’t really know whether it understands you or not. And chances are it doesn’t, at least not as much as you feel it does.
Stumbling blindly over its own shortcomings, without ever taking responsibility…
If you think that it’s a bad situation when you’re sitting for coffee with a friend, or for a second coffee at your computer screen with an AI, think about what might happen when these AIs with limited understanding of you become agentic and perform tasks for you. Everything seems to be running perfectly fine until there’s a massive blunder in an area that this AI should have really gotten right, given how much it demonstrated its understanding before. Like if your therapist helped you out in-person in that argument with your spouse, beautifully, only to ruin the whole thing by getting their favorite color wrong!
Worse yet, it’s you who has to live with the consequences, not the AI. The AI (or rather, its producer) just collects your bill at the end of the month.
As a result, when you’re working with AI, it’s really crucial to double-check every step, and that can be a very tedious process unless it’s architected right. One of the simplest approaches one can do is have different AIs check one another’s work. This is very efficient because AI is fast, but it’s not sufficient.
What we really need is a proper way of imbuing AI with “thinking.” And luckily, a mathematically rigorous formulation of “thinking” already exists.
Causal inference is not a magic brain update — but it’s quite the A-Eye-Opener.
That method is called causal inference. It was actually developed by mathematician Judea Pearl decades ago, but it’s becoming more and more in vogue because it turns out that it’s a brilliant technique to check and guardrail AI.
It’s really a set of methods for estimating the effect of a defined change of some variable under some kind of stated assumptions. So for example, A changes by 10% and A somehow depends on B. So how does B now change? This is the causal link.
It also asks what counts as an intervention and what else is being affected around that link between A and B, what an outcome C could be of all that, and whether variable D could also influence what’s happening between A and B.
Furthermore, once a causal link is established, one can simulate interventions and scenarios. What now happens to outcome C if I change A a little bit, or if I add something to D? Humans like to optimize for good outcomes, and causal inference provides the mathematical underpinnings to do just that.
Of course it’s a bit more complex than that (it’s math, after all) — but this formalism, at its core, provides very solid conclusions that resemble human thinking, augmented by a level scientific rigor that most human minds still aspire to. Stack that into AI and you truly have a form of superintelligence, and not just great-sounding gibberish.
Causal inference transfers responsibilities better, through enhanced system understanding.
An interesting corollary of this whole interventions and scenario running in causal inference is that it can help AI understand what consequences are and how to live with them. So we humans are not all alone in facing and bearing the consequences of our or increasingly AI’s actions.
In practice, this would look something like this:
First, before even touching the data, an AI agent inspects the dataset itself, the questions, and formulates some hypothesis. Something like, “I think that A causes B and therefore we’d get outcome C.”
It would then draw up that hypothesized system as a second step. This produces a causal map of sorts where you can see all the causes and outcomes and everything in between in one chart.
Then it would orchestrate approved analysis tools rather than, as is the case at the moment, mostly relying on its own trained logic to solve math problems (even when its logic is wrong!).
It would also make abstention an achievement. That is, if the AI is not sure, then it would honestly say so, and it would win brownie points for doing so. Right now, AI is conditioned to say something that sounds knowledgeable, even when it actually has no idea what it’s talking about.
Fifth, it would then generate some actionable advice and, if applicable, execute on it. But it would only execute (and thus be a real agent) if it is authorized to do so by a human. That gate is a friction point in the system, which is really important. A bit of friction and human control is a good thing at this moment in AI development.
And finally, this AI should learn by itself from the outcome and log whatever it’s been doing and how it got to that outcome. This way it can review the outcome every so often and optimize for the best outcome.
This gives AI a clear stake in what the outcome actually is and gives it some incentive to optimize towards the outcomes that humans really want. That’s more than just generating cool sounding gibberish that, at the end, some human not only has to pay token fees for, but also needs to sign with their name.
The status quo: Somewhere in the muddy grounds between “sci-fi” and “solved.”
Sounds like a cool workflow, right? In fact, this is exactly what we are implementing in our flagship product, Etio, and our other products, Capvert and Forecaus.
However, looking at the research and the status quo out there (consider this, this and this), what I’m seeing is that humanity is still somewhere in the middle of this journey. The conceptual frameworks of how to actually deploy causal inference within AI within enterprise systems are still developing. And so these problems are far from solved — but they are also not sci-fi anymore.
If you told me about all of this two years ago, I would have said, “woo, dude, that’s not happening.” And now it is happening! We are still very much in the beginning of this journey — it’s far from solved — but it’s definitely happening, and not just at Wangari.
That said, I’ve yet to see some real use cases where this system just runs in a scalable way without any hiccups whatsoever, like a good laundromat. Nevertheless, we’re inching closer and closer pretty fast.
If something doesn’t work, don’t try harder. Especially thinking.
Most people can’t just open up an AI, look inside, and give it a framework to help it “think.” Most people can’t string together multiple AIs, and plonk them in a framework that does causal inference. And neither of this I’d demand of you readers. But if there’s one takeaway for you in your day-to-day work, I would say: Don’t make AI think too much, and don’t believe too much what it says it thinks.
Research has shown that what AI tells you it’s thinking and what it’s actually thinking are two very different things. And so, if you’re not satisfied with an answer, it’s of limited value to ask the AI, “hey, why did you come up with this?” The AI likely doesn’t even know why and how it came up with some result — and even if it does, it’s not incentivized to tell you all about it.
Instead, try simplifying your approach or your prompt. When thinking harder doesn’t work, think less. This, at this moment, is the safest approach. Meanwhile, firms like Wangari work in the background to strengthen AI — so that you get better and better outputs on your end of the screen.
Meanwhile, at Wangari
I’ll be speaking at the Erfahrungsaustausch Solvency II in Leipzig this Thursday — getting down to core principles on how to use AI in insurance reporting without getting crucial things wrong (and crucial things are all things in reporting, by the way; every number or sentence could make or break the report).
The broader event has two main pillars: Automation (including AI), and handling the upcoming Solvency II Review, which — supposedly — eases the reporting burden. My talk straddles both because I see in this regulatory change an opportunity for AI to add value in unprecedented ways. More to come when I’m back from Leipzig.
Reads of the Week
AI Engineering Insider has an excellent piece explaining why AI’s “thinking mode” doesn’t always produce better results, despite being slower. The guide also explains how AI thinking is implemented under the hood, which is very valuable for anyone tinkering with AI for a living (me, and many of you). Moral of the story: Easy tasks don’t benefit from extra thinking, complex ones do — and the wise user knows the difference between the two.
A fascinating journey into AI for medicine, by Physician Logic Squared: OpenAI’s Epic now not only knows the world’s publicly available knowledge of medicine, but also countless patients’ records. However, as the author points out, AI is not able from this to construct causes and effects of various diseases, or prescribe a treatment. This should be done by a doctor, whose role increasingly will be curating and editing AI-generated answers to disease, and making the final judgement.
Arnaud Blandin makes the case that AI should enable humans to return to more human work — you know, the kind that happens face-to-face and not behind a screen. He then goes into the human work that many companies are doing to make the planet a better place, and the Shizenso methodology he created around streamlining such work. It’s an interesting thought-starter on where AI fits in corporate governance and sustainability.