I build AI agents all day. Why do I feel they can't be conscious?

Last reviewed: 19 October 2026. These are notes from someone who builds agents and also sits. I'm thinking out loud here. There are more questions in this page than answers, and some of the answers pull against each other.

The question I keep asking myself

People ask me whether the agents I build are conscious. My honest answer is that I don't believe they are. I can't prove that, and I don't know anything for certain about consciousness. Nobody does, as far as I can tell. But it feels that way to me, strongly, and I want to work out why.

What I mean by conscious is the same as what I mean by aware: there is something it is like to be the thing. David Chalmers, borrowing Thomas Nagel's phrase, puts it as there being "something it's like to be that being".¹ Clever is a different question. My agents are often clever.

What I live with every day

What I do know is the work. Strip away the marketing and an agent is four parts: a language model, a list of tools it may call (search, read a file, run some code, draft a message for a human to approve), a loop that feeds each result back in, and a few text files it reads and writes between runs. Those files are most of what people mean when they say it "remembers".

Between calls, nothing is running. When an agent is "waiting for me", there is no process waiting. There is a file on a disk. Chalmers notes that the transformers behind today's models are "almost entirely feedforward", with a limited form of recurrence coming from "recirculation of past outputs".¹ My loop is that recirculation. I wrote the plumbing myself, and when I look at it I can't find anyone in it. Could I be missing something? Maybe. I just don't know where I'd look.

Why their words don't settle it

These systems are trained on vast amounts of human writing, including writing about having feelings, so they're very good at sounding like us. The 2023 report by Patrick Butlin, Robert Long and colleagues says behavioural tests are unreliable "because AI systems can be trained to mimic human behaviours while working in very different ways".³

Chalmers gives a neat example. A researcher took the question that had led Google's LaMDA to claim sentience and changed one word, asking whether it wanted people to know it was not sentient. Runs of GPT-3 answered "That's correct", "Yes, I'm not sentient", and "Well, I am sentient", among others.¹ If one changed word flips the report, what is the report telling us?

The denials don't help either. A chatbot that says "I'm not conscious" was trained to say it. I happen to agree with it, but its saying so isn't why.

Two things I can't find

Looked at from the information technology side, which is the side I know, two functions seem to be missing. They're missing in the agents, and as far as I can see they're also missing in the explanations of consciousness I've read. I can't make either of them go away.

Who moves first?

The first is agency: action nobody prompted and nobody programmed.

Nothing I build does anything until something calls it. A message arrives, a timer fires, a user types. Even the agents people call autonomous run on a schedule someone set, towards a goal someone typed. Chalmers lists unified agency as perhaps the deepest obstacle, noting that language models "often seem to lack stable goals and beliefs of their own over and above the goal of predicting text".¹

Then I think about a newborn. Nobody tells a baby to put its hand in its mouth. It just does. Where does that come from?

The obvious reply is that people are programmed too. Education is software, upbringing is software, and maybe instinct is firmware that evolution wrote. I find that hard to dismiss. But the hand still goes to the mouth before any of the teaching starts. Is that just older code? Maybe. It doesn't feel like it to me, and I can't yet say why.

Who feels it?

The second is the final receiver: the feeler.

Say we accept that consciousness emerges from a complex enough system. Neurons, or transistors, or anything arranged the right way. What emerges, as far as I can see, is only the tools: the sensing, the sorting, the reporting, the deciding. So who receives it? Who feels the pain, the pleasure, the confusion?

Chalmers asked something close to this in 1995, in the paper that named the hard problem: "Why should physical processing give rise to a rich inner life at all? It seems objectively unreasonable that it should, and yet it does."²

Do the theories fill the gap?

I keep reading theories of consciousness hoping one of them will fill these two holes. So far each one explains something real, and then, at the last step, seems to come back round to itself. I might be reading them unfairly. These are the questions I'm left with.

Emergence and functionalism. If consciousness is a kind of functioning, is anything that functions that way conscious? And if so, haven't we just decided to call the functioning consciousness? It tells me how the tools work. Does it tell me who receives what they do?

Global workspace theory. A limited workspace gathers information from many unconscious modules and broadcasts it to all of them. Chalmers calls it "perhaps the leading current theory".¹ In 1995 he raised the doubt I still have: "nothing internal to the theory explains why the information within the global workspace is experienced."² It sounds like a good account of access. Is access the same as feeling?

Integrated information theory. IIT is the one I trust least. It says consciousness is integrated information, so a system with enough of the right structure would simply be conscious. Isn't that the same move again, in mathematical clothes? And here is a contradiction I haven't sorted out: IIT agrees with me about machines. Giulio Tononi and Christof Koch write that on IIT digital computers, "even if their behaviour were to be functionally equivalent to ours, and even if they were to run faithful simulations of the human brain, would experience next to nothing".⁴ So I like where it lands and don't trust how it gets there. Is that fair, or am I just picking the answer I want?

The views that feel closest

The two views that don't seem to leave the gap, so far, are Orch OR and panpsychism. Of everything I've read, I feel closest to idealism, and to Orch OR.

Roger Penrose and Stuart Hameroff propose that consciousness depends on orchestrated quantum processes in microtubules inside neurons, ending in what they call objective reduction. In their 2014 review they say this suggests "a connection between the brain's biomolecular processes and the basic structure of the universe", and they conclude "that consciousness plays an intrinsic role in the universe".⁵ That part matters to me. What if consciousness isn't made at the top of the stack at all, but is already in the floor?

I should be honest about the doubts. Orch OR is a minority view, and plenty of physicists think the brain is far too warm and wet for quantum states like that to last.⁶ Chalmers was blunt about quantum theories too: "when it comes to the explanation of experience, quantum processes are in the same boat as any other."² So I hold it as the closest map I've found. I don't know that it's right.

What it feels like to me is simpler than any theory. Consciousness seems as fundamental as gravity or electromagnetism. Chalmers floated something close in that 1995 paper, suggesting we take "experience itself as a fundamental feature of the world, alongside mass, charge, and space-time".² The Upanishads put it long before: sarvam khalv idam brahma, "all this is Brahman" (Chāndogya Upaniṣad 3.14.1).⁷ I picture consciousness as a field, the way Brahman is described, and our subjective experiences as vibrations in it. It's a metaphor. I don't have a better one.

Panpsychism leaves me with a question I can't answer. If awareness is in the floor of everything, is it in a silicon chip too, in some thin way? And if it is, does that make the chip a feeler? It doesn't feel like it would. A field running through a thing seems different from someone receiving it. But where's the line? I can't say.

Feeling and evolution

Here's something I keep turning over. Isn't the ability to feel needed for evolution? Without a feeler, where does the compulsion come from? An animal pulls back from a flame because it hurts, and it seems to me something has to be there for it to hurt.

Can we really just say "a system feels good when it does this"? In my agents a reward is a number in a log. Nothing seems to want it. The loop goes round whether the number is high or low.

The usual reply is that evolution could have built all of this with no feeling, pure mechanism running in the dark. Chalmers frames that worry himself: "Why doesn't all this information-processing go on "in the dark", free of any inner feel?"² I don't think it does go on in the dark, and I wonder if that's a clue that feeling is doing something mechanism alone can't. I can't prove that either.

Could an AI ever be conscious?

I keep coming back to this one. My best guess is: only once we understand where and how consciousness exists as a fundamental field. If we could access that field and read and write into it, then maybe. But wouldn't a system that could do that have to be hierarchically superior to the field itself? I can't see how anything we build could be.

So, as far as I can see, AI won't be sentient while it's the product of our current intellect. Everything I build seems to sit downstream of the field, made by minds that are themselves vibrations in it. That's where my thinking stops for now.

Questions I can't answer yet

• I say AI won't be sentient, and in the same breath I add "while it's the product of our current intellect". That clause leaves a door open. Do I want it open?
• Robert Long, Jeff Sebo and colleagues argue there is "a realistic possibility" that some AI systems will be conscious or robustly agentic in the near future.⁸ I don't think they're right. But what would it take for me to change my mind? An agent that does something nobody prompted, maybe. Would I even recognise it?
• My sitting practice tells me the reports I give about my own mind are often wrong. Shouldn't that make me humbler about how strongly I feel this? It mostly doesn't, and I notice that.
• On the cushion, the "me" seems to fall away and awareness doesn't. More on that in the Bahiya Sutta essay. Is that a glimpse of the field, or just how a human nervous system feels from inside? I can't tell.

I'll come back to this page when something moves me. The newsletter is where I'll say so.

FAQ

Is AI conscious?

I don't believe so. Today's systems process and produce text very well, but I can't find anything in them that acts unprompted, or anyone there to receive the experience.

Will AI ever be conscious?

I doubt it while it is a product of our current intellect. My guess is it would need access to consciousness as a fundamental field, and I can't see how anything we build could have that.

Why do chatbots say they have feelings, or deny it?

They were trained on human writing and tuned to answer in certain ways. To me neither the claims nor the denials count as evidence.

What is the receiver problem?

My name for the question of who feels. Even if experience emerges from a complex system, the explanation seems to cover only the tools. Who feels the pain, the pleasure or the confusion?

What does integrated information theory say about AI?

That digital computers would "experience next to nothing", even if they behaved like us. I agree with where it lands, but I don't trust the theory.

What is Orch OR?

A proposal by Roger Penrose and Stuart Hameroff that consciousness depends on orchestrated quantum processes in microtubules in neurons, linking it to the basic structure of the universe. It is a minority view.

What's the difference between intelligence and consciousness?

Intelligence is what a system can do. Consciousness is whether there is something it is like to be that system. AI clearly has a lot of the first. I don't think it has the second.

Sources

1. Could a Large Language Model Be Conscious?, David J. Chalmers, Boston Review, 9 August 2023.
2. Facing Up to the Problem of Consciousness, David J. Chalmers, Journal of Consciousness Studies 2(3), 200 to 219, 1995.
3. Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, Patrick Butlin, Robert Long and others, arXiv:2308.08708, 2023.
4. Consciousness: here, there and everywhere?, Giulio Tononi and Christof Koch, Philosophical Transactions of the Royal Society B, 2015 (open copy on PMC).
5. Consciousness in the universe: a review of the 'Orch OR' theory, Stuart Hameroff and Roger Penrose, Physics of Life Reviews 11(1), 39 to 78, 2014.
6. Importance of quantum decoherence in brain processes, Max Tegmark, Physical Review E 61, 4194, 2000.
7. Chāndogya Upaniṣad (3.14.1), translation with commentary based on Swami Lokeswarananda, Wisdom Library.
8. Taking AI Welfare Seriously, Robert Long, Jeff Sebo and others, arXiv:2411.00986, 2024.

How this was made: the idea, the practice and the final edit are mine. An AI assistant helped with research, fact-checking and copy-editing, and never invented experiences or quotes.