I’ll start with a sentence that I say often, falsely. I often say something like, “the model returned the result XYZ.” But that’s not true.
A model doesn’t wake up, inspect company data, decide which systems it can access, call an API, check whether numbers are anomalous, remember yesterday’s exception, or ask a human for approval. The agentic system does those things, or is configured to do them through a full-blown architecture around the model.
So actually the model didn’t return much. It’s the whole system that did something that enabled the model to spit out a final result.
The model isn’t the whole agent. It’s just one component in a loop.
When we say that an AI agent acted, we usually mean a great deal more than “a model generated a sentence.” (Although that’s of course a key piece of it.) We mean that it had a view of a changing world, some memory of what had happened before, tools it could use, limits it could not cross, and a way of verifying its results. In other words, it had something closer to an environment.
To put this, carefully, into a human analogy: Did I write my PhD thesis, or was it rather an automatic result of my natural tendencies inside an environment containing a university, research lab, and PhD supervisor? Did “I” really do the writing or was it rather my hands (this was pre-LLM-era), as directed by my mind, aka “the model”?
Humans of course are not disembodied language models, so this is deliberately said carefully. Conscious life – whatever that is – happens in a living system that is continuous with a physical body, a mind, and a surrounding world. Physical bodies in one way inform about the border between the world and our surroundings, and also indicate the contents of our mind (by reacting to our thoughts and emotions in physically apparent ways).
At a minimum, human minds appear inseparable from an organism that senses, acts, persists, regulates itself, carries a history, and is continually changed by the consequences of its actions, both physically and mentally.
So: if we assume that human consciousness happens in a biological body (I have yet to meet a conscious disembodied ghost, which we’d arguably not call quite human anyway), then what would count as a “body” for an AI agent?
And if an agent ever became more than an extremely capable tool, and something with consciousness, would that transition happen inside a neural network or rather in the ongoing relationship between the model, memory, sensors, tools, constraints, and the world?
I find this an extremely intriguing question because in deepening my meditative practice, I start to see that enhanced consciousness comes from withdrawing from identification with the senses, the body, and the contents of the mind. Not to say “not feeling, not thinking” — quite the opposite — but neutrally observing physical sensations, emotions, and thoughts, without getting caught up in them.
But at the same time, my research around AI pulls me in the other direction: AI seems to need some form of embodiment. In software, that embodiment is akin to its environment and its architecture (not some kind of physical robot body). My suspicion — spoiler alert — is not that agentic architecture is a route to machine consciousness per se. It is that agentic architecture may be the only place where that question could become meaningful, and that’s worth unpacking in a deeper way.
Ingressing Minds
Now this musing doesn’t come from my own work. Michael Levin recently published a paper called Ingressing Minds, and he makes an argument that will sound either exciting or intolerable, depending on your stance on metaphysics.
But first, credits where credits are due: I didn’t come across that paper by myself, but by means of Jack Clark’s brilliant Import AI newsletter.
Back to the subject: Levin argues that the patterns responsible for form, function, and agency may not be fully explained in physical material or algorithms alone. Levin proposes a framework in which bodies – living, engineered and hybrid – act as interfaces through which a hierarchy of patterns can become effective in the physical world. He places agency on a spectrum from simple fixed patterns and goal states to behaviors recognizable as minds.
In other words, I might just be some kind of evolutionary supercomputer whose fingers type on this keyboard in a predefined sequence. Which may be true, but I’d contest that (a) I have a subjective experience of consciousness and (b) the pattern of my finger movements on this keyboard is complex enough to outmode an AI.
Levin’s thesis is not conventional neuroscience or the result of a result that anybody should treat as established. It’s a proposal about ontology, a suggestion about what kind of things might be real and causally relevant.
But the engineering question hidden inside this is much more tractable: What’s the relationship between an architecture and the kind of agency that becomes possible within it?
Levin argues that machines and organisms should be considered along a spectrum and not treated as categorically separate just because one is built and the other evolved over millions of years. So he explicitly includes robotics, software AIs, and language models in a call for humility about engineered constructs.
His reason is not that they obviously have minds or something. It’s that builders can these days create systems whose organized capacities exceed what they even understand. In other words, we know that superintelligence is already happening and all around us. So we should be humble about our place as Intelligence on Earth.
Levin’s claim is much stronger than the one that I need to make. I don’t need to believe in platonic forms, for example, which is a key component he draws from, or ingressing minds, or robot souls. I just need to accept the narrower warning, which is that building a system doesn’t guarantee that we understand the level at which its behavior is organized.
A model is not an agent
Now here’s where consciousness conversations about AI often become unhelpful. They focus on benchmarks, outputs, chain of thought looking text, or whether a model says the words, I feel. They imagine that a model is some kind of isolated object, like a brain in a jar that happens to speak.
But first of all, humans aren’t brains in jars for the most part. And to put this into an enterprise context, an enterprise agent is also not just a text generator standing alone, it’s a control system embedded in an organization. The language model is often the part that speaks. It’s not necessarily the part that has a stable relationship to the world.
My mind doesn’t know the world all that well. It’s my hands that really know the keyboard expertly well, and my eyes that know the screen. A model can produce a beautiful account of an objective without possessing an actual objective. It can describe a world without being answerable to one.
Now here’s how an enterprise agent becomes a continuing system, not necessarily conscious, but continuing:
Perception: access to data feeds, documents, events, and signals from the organisation.
A maintained world model: an explicit representation of entities, relationships, current state, uncertainty, and context—not merely a transient prompt.
Memory: the ability to retain history, commitments, exceptions, and lessons across time.
Tools and effectors: systems it can query, actions it can propose, workflows it can initiate, and changes it can make.
Goals and evaluation: a measurable or at least inspectable account of what success, failure, risk, and escalation look like.
Feedback: evidence about whether the action had the intended effect, including correction by people and by the environment.
Boundaries: permissions, policy constraints, logging, approvals, and conditions under which the system must stop or hand work back to a person.
Persistence: an identity that survives one chat window and one task, with a history that is not reset by every interaction.
None of these items is consciousness, but together they produce something that’s much more like an organization’s, an organism’s ongoing relationship to a world than a prompt response interface has. We shouldn’t call that consciousness, of course, but we should stop pretending that the consciousness question is exhausted by inspecting a model’s output, at the very least.
Now, permissions layer is of course not a nervous system like those that humans have. An API is not a hand, a database is not a memory in the human sense either. Analogies become dangerous when they become camouflaged for a conclusion.
The point I’m making is architectural and not biological: An agent becomes consequential through the closed loop between an observation, an internal state, an action, and feedback. And that loop is what makes the question so interesting.
How this ties to causal structure
Now comes the part where the conversation stops being metaphysical decoration and becomes an infrastructure problem. An agent operating in an organization shouldn’t just produce plausible language. It should be able to represent the relevant system, distinguish observation from intervention, track uncertainty, and expose why it recommends a specific action.
If an agent is going to act in the world, it needs more than a vocabulary for the world. It needs a model of consequences. Now, a quick distinction here. Causal structure isn’t consciousness. A causal model is no proof that anything is experiencing the system’s actions, but causal structure does matter for agency.
Because it lets the system orient action around plausible interventions and measurable consequences rather than superficial patterns in historical data. Correlation lets the system notice that two things move together. Cause or structure is what lets it ask, if we change this, what follows automatically? And how would we know?
That’s not a consciousness test, it’s a responsibility test. But the question that a regulated organization has to answer is also not, does the agent feel? It’s what did it believe? Why did it act? What could it affect? And who can reconstruct the answer?
And so that’s where these thoughts do get become very valuable. For a software agent, infrastructure is not a deployment detail, it’s the condition of its agency, its data environment. Determines what it can perceive. Its semantic and causal model determines what it can make sense of.
Its tools and permissions determine what it can affect. Its feedback loops determine whether it learns or just repeats like a pirate. If there is a meaningful analog of a body here, it’s not in a humanoid shell, it’s in this. Surrounding architecture, the interface through which a system encounters a world and makes a difference to it.
As an aside, this is partly why I find the current fixation on the model itself so odd. In production, the model is often the least durable part of the system. The architecture is where the commitments live.
Maybe metaphysics isn’t the right argument for embodiment?
Now, there’s a couple of objections to this whole argument, and I wouldn’t want to spare those from you. The first one is that functionality is not the same as experience. A system can have perception-like inputs, memory-like storage, goal-directed behavior and feedback loops without having any inner life whatsoever. For example, a thermostat is not a person. Scaling up the architecture doesn’t by itself resolve the hard problem of experience.
However, this may evidence that our old categories — model versus tool, software versus agent, intelligence versus just automation — are a bit too blunt for the systems that we are now building. Perhaps some finer distinctions are at the ordre du jour.
The second objection is that we shouldn’t be anthropomorphizing systems that can’t be held accountable. Of course, calling agents conscious too early could make people overtrust them, excuse organizational failures or blunt human accountability. An organization can’t outsource responsibility to a system simply because it behaves with an Nerving coherence, traceability, bounded permissions, explicit uncertainty, and the ability to challenge an output aren’t obstacles to a system’s agency. They’re the conditions under which agency is legitimate in an enterprise setting.
And that does translate to human behavior. So that’s where this kind of thinking again becomes useful.
The third objection is that Levin’s papers, Metaphysics and Enterprise, need evidence. Levin’s framework may ultimately be wrong. The claims about non-physical patterns may be untestable or at least not earn their keep against more conventional explanations. This is possible, but the value of the paper goes beyond that. It’s not methodological as such. It tells engineers to be humble about novel systems, to test what they’ve made, and not to mistake a construction for a complete explanation.
This becomes more and not less important when agents are trusted with consequential work.
Placing responsibility before consciousness
So, is agentic architecture the way to AI consciousness? I don’t know. I don’t think anybody else knows either.
But if machine consciousness is ever more than a philosophical parlor game, it’s unlikely to arise from a disembodied text model viewed in isolation. It would have to involve something enduring, some structured relationship between a system, a world, its actions and the consequences of those actions. This relationship is exactly what we’re now building when we connect models to enterprise data, tools, memories and decision workflows.
We don’t need to call those systems conscious to take their architecture seriously. We do need to make them legible, bounded, contestable and accountable before they’re entrusted with more of the world. So off camera, you may want to entertain your thinking about conscious systems while, you know, phrasing it more responsibly when you actually go and do the work.
The question isn’t whether agents will become human. Probably not. Probably they’ll become superhuman or something entirely different. The real question is whether we’ll build them with enough understanding to remain responsible for whatever they do become. Consciousness may be the distant question here. The immediate one is architecture.
Either way: before we ask whether an agent has a mind, we should make sure that it has a world model that we can inspect, and guardrails of how it may interact with its surroundings.
Meanwhile, at Wangari
Next week is the last edition before another two-week break. Not for meditation, this time (though that’s part of my daily life anyway) — I’m traveling in Europe to see some family, and would like to focus my mind on strictly human matters during this time.
Reads of the Week
If you’re still asking “WTF are AI Agents?” — you’re not alone. Thomas Paule to the rescue! In a very practical way, he relays what makes AI agents so special — “building an agent is like building a human.” And how to get started, which, for most people, is Claude Cowork, Copilot, or Codex/ChatGPT.
AI Consciousness and the Embodiment Problem — a very clear and opinionated book review by Josh Gellers, PhD. Somehow, consciousness and embodiment do go together, but to claim that you need a body to be conscious or to blindly assume that academic research says AI is not conscious so it’s not — that is a bit quick. Gellers picks this apart in a very clarifying way.
Under what conditions would a machine be conscious, and could that be tested? That’s the question CIMC asks in publishing a 26-page essay which is worth a Sunday afternoon (the essay is linked rather than the body of the Substack piece). With my own background in Indian meditative practices, it’s quite interesting to see their distinction between Western views and views of other cultures on AI, which, in many ways, seem much more advanced.