What has already been said about consciousness and AI? Part 3: The machines

Series: Part 3 of 3 in the Machine and mind essays.
This part covers the machine question itself: the Turing test, the Chinese Room, Penrose's argument, large language models, the welfare debate, what people believe, and the questions I was left with. It builds on Part 1, the old puzzle, and Part 2, what the scientists say.

Has AI passed the Turing test, and what does that show?

Everything in Parts 1 and 2 was background for the question I actually started with. Can a machine be conscious?

Alan Turing's 1950 paper opens with almost that question. His first sentence proposes to consider the question "Can machines think?" A few pages later he calls the original question "too meaningless to deserve discussion" and replaces it with a game (Turing 1950). An interrogator chats in text with a person and a machine and tries to tell which is which. Turing even expected a good machine to play tactically. He wrote that it "would deliberately introduce mistakes in a manner calculated to confuse the interrogator". He also made a prediction: by the end of the century, he wrote, people would be able to "speak of machines thinking without expecting to be contradicted". I had always thought of the test as a test of mind. Reading the paper, I found it is a test of imitation, and Turing says so.

That matters now. In 2025 Cameron Jones and Benjamin Bergen ran three-way text conversations in which people chatted with a human and an AI at the same time. GPT-4.5, told to adopt a persona, was judged to be the human 73% of the time, more often than people picked the actual human. In their preprint, Jones and Bergen call this "the first empirical evidence that any artificial system passes a standard three-party Turing test" (Jones and Bergen 2025). Turing himself had swapped "Can machines think?" for a game of imitation. So if the game can be won, what does winning it show? I kept turning that over.

Can a program understand? The Chinese Room and the Gödel argument

John Searle's Chinese Room, from 1980, is the best-known argument that passing would never be enough (Searle 1980). Imagine someone who speaks no Chinese, locked in a room with a rulebook. Chinese questions come in, and by following the rules the person sends back fluent Chinese answers. From outside, the room understands Chinese. Inside, nobody does. Searle's conclusion is that shuffling symbols by rules, which is what a program does, is never enough for meaning.

The Stanford Encyclopedia lists many replies (SEP, Chinese Room). The two I met most were the systems reply, which says the person does not understand but the whole room does, and the robot reply, which says give the system a body and senses so its symbols connect to the world. Forty years on, people are still arguing about it.

There is a second, quite different argument that machines cannot fully understand, and I followed it separately because it comes from mathematics. In 1961 the philosopher J. R. Lucas argued that Gödel's incompleteness theorem shows the human mind is not a machine (Lucas 1961). Roger Penrose took the idea up in The Emperor's New Mind (1989) and its sequel Shadows of the Mind (1994). Penrose argues that human mathematical understanding is non-computational. Roughly, for any formal system we could know to be sound, there are truths we can see that the system cannot prove (Penrose 1989; Penrose 1994; Megill, IEP). If he is right, no program could understand mathematics the way we do. That is why he goes looking for new physics, and ends up with Orch OR, which I covered in Part 2.

The replies were the part I could check most carefully. The Internet Encyclopedia of Philosophy explains that the argument fails if human minds are inconsistent, or if we cannot know that we are consistent, and notes that Hilary Putnam put this to Lucas in conversation even before 1961 (Megill, IEP). Putnam's own review of Shadows of the Mind says the argument is one that experts in mathematical logic "have long rejected as fallacious" (Putnam 1995). Chalmers wrote that "the greatest vulnerability in this argument lies in the assumption that we know (unassailably) that we are consistent" (Chalmers 1995b). The logician Solomon Feferman found Penrose's discussion of Gödel "marred by a number of errors" (Feferman 1995). Grush and Churchland put the alternative in one line: "unsound, albeit reliable, algorithmic processes escape Gödel's net" (Grush and Churchland 1995). Penrose replied to his critics in 1996. I have only seen that reply cited, not read it.

Is there a test for AI consciousness?

If behaviour is not enough, what else could you test? The zombie argument from Part 1 is the worry in the background.

Susan Schneider and Edwin Turner proposed an AI Consciousness Test. It asks whether a system can grasp ideas that seem to rest on having experience, such as swapping bodies or surviving death (Schneider and Turner 2017). The catch is that it only works if the AI has been kept away from human writing about consciousness. Today's chatbots have read nearly all of it. In 2023 Schneider accepted this and added that we need to look at what is happening inside the system, not only at what it says (Schneider 2023).

Murray Shanahan, one of the scientific advisors on the film Ex Machina, put a name to a different idea. In an interview he recalls a line in which Nathan tells Caleb that the point is to show him that Ava is a robot and "see if you still feel she has consciousness". Shanahan thought the line was spot on, and named the idea after Alex Garland, who wrote the screenplay: "So I call that the Garland test" (Shanahan interview). You show someone plainly that they are dealing with a machine, and then see whether they still feel it is conscious. It measures our response, not the machine's inner life. I found that oddly honest.

Could a large language model be conscious?

The most systematic attempt I found to judge actual systems came from Patrick Butlin, Robert Long and seventeen co-authors in 2023. They drew 14 indicator properties from scientific theories, including several from Part 2, and checked current AI against them. They write that their analysis "suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators" (Butlin and others 2023). The whole approach assumes computational functionalism, the idea that the right computation is enough. If that assumption is false, the checklist tells us much less. A later update of this work exists, which I have only seen as a record.

Chalmers tried putting rough numbers on it in 2023. He weighs the obstacles current models face, among them the lack of recurrent processing, of senses and a body, and of unified agency. He ends up with "somewhere under 10 percent" confidence that current models are conscious, and 25 percent or more that more complete successors will be within a decade (Chalmers 2023). He warns more than once not to take the numbers too seriously.

In 2022 a Google engineer, Blake Lemoine, went public with his belief that the company's LaMDA chatbot was sentient. "If I didn’t know exactly what it was, which is this computer program we built recently, I’d think it was a seven-year-old, eight-year-old kid that happens to know physics," he told the Washington Post (The Guardian, June 2022). Google later said his claims were "wholly unfounded" and dismissed him (The Guardian, July 2022). The episode settled nothing about LaMDA. It did show how easily fluent talk persuades.

Before that, the best-known sceptical paper had come out in 2021. Emily Bender, Timnit Gebru and colleagues described a language model as a system for stitching together sequences of linguistic forms from its training data, according to probabilities, "without any reference to meaning: a stochastic parrot" (Bender and others 2021). Their wider point was about costs and harms, and about what follows when fluency is treated as understanding.

Shanahan, Kyle McDonell and Laria Reynolds offered another frame in Nature. Think of a chatbot as playing a role. That lets us use everyday words about its behaviour "without ascribing human characteristics to language models that they in fact lack" (Shanahan and others 2023). Their two test cases are apparent deception and apparent self-awareness.

The newest work I read asks a narrower question. Anthropic's 2025 introspection research tests whether models can accurately report on their own internal states. Its own FAQ is careful to say this does not speak directly to phenomenal consciousness (Anthropic 2025a).

What is AI welfare, and why are some researchers taking it seriously?

Here the debate turned in a direction I had not expected. AI welfare, as I met it, is work on whether AI systems could have wellbeing or moral status, and on what to do while nobody can tell. Some researchers argue that we may never get a clean answer, so we should work out how to act without one.

The model came from animals. The New York Declaration on Animal Consciousness, from April 2024, says that where there is "a realistic possibility of conscious experience" in an animal, "it is irresponsible to ignore that possibility" (New York Declaration 2024). Jonathan Birch's book The Edge of Sentience (2024) builds a precautionary framework for uncertain cases. He also names the gaming problem: a language model trained on human text can reproduce the signs we would look for, so its words about its feelings are weak evidence. He concludes we need "deep computational markers" inside the system (Birch 2024).

Robert Long, Jeff Sebo and colleagues applied this to AI in Taking AI Welfare Seriously. They argue there is "a realistic possibility that some AI systems will be conscious" or robustly agentic in the near future, and that AI companies "have a responsibility to start taking it seriously" (Long and others 2024). Long now runs Eleos AI Research, a nonprofit working on the wellbeing and moral status of AI systems (Eleos AI). In April 2025 Anthropic announced its own model welfare research, saying "There’s no scientific consensus on whether current or future AI systems could be conscious" and that it would approach the topic "with humility" (Anthropic 2025b).

Not everyone in the industry agrees. In August 2025 Mustafa Suleyman wrote about what he calls "Seemingly Conscious AI", systems that show all the outward signs. He says there is "zero evidence" of AI consciousness today, and that the urgent risk is people coming to believe in it. His title states his view: "We must build AI for people; not to be a person" (Suleyman 2025).

What do AI researchers and the public believe about AI consciousness?

I wanted to know how far opinion had moved, so I looked at surveys.

Among professional philosophers, the 2020 PhilPapers survey, as reported by Chalmers, found about 3% accepting or leaning towards the view that current AI systems are conscious, and about 39% for future AI systems (Chalmers 2023).

A 2025 survey by Noemi Dreksler and colleagues asked 582 AI researchers and 838 members of the US public. The median researcher put the chance of AI with subjective experience existing by 2034 at 25%, and the median member of the public at 30%. By 2100 the medians were 70% and 60% (Dreksler and others 2025).

Clara Colombatto and Stephen Fleming asked 300 US residents about one language model, ChatGPT. About two thirds (67%) attributed some possibility of phenomenal consciousness to it, and those who used it more often attributed more (Colombatto and Fleming 2024).

A survey measures what people believe, which is a different thing from what is true. Still, belief shapes how these systems are designed and regulated, and how they are treated, so I do not think the numbers are beside the point.

Where my reading left me

No theory has won. The biggest head-to-head test so far challenged both of the theories it tested, and every view I read, from dualism to Orch OR, has a serious criticism still standing (the summary table below sets them side by side).

What struck me is that almost every verdict about AI depends on one earlier choice. Is the right computation enough? Or does consciousness need biology, life, a particular physical structure, or something more basic still? Once someone has made that choice, their answer about machines mostly follows.

Behaviour can no longer settle it, either. Jones and Bergen report that the Turing test has now been passed, and the gaming problem means that what a model says about its inner life may be borrowed from us.

The welfare question has moved too. It is less "is it conscious?" and more "how should we act when we cannot tell?"

This is where I got stuck. What would count as evidence for me? And is a mistake in one direction, treating something that feels as if it were empty, worse than a mistake in the other?

That is where the series ends, with more questions than I started with. If you came in here, Part 1 starts with the old puzzle and Part 2 with the science. As someone who builds AI agents, I have put my own felt view in a separate essay. There is also an older piece of mine on a 2,500-year-old instruction: in the seen, only the seen.

Sources

1. Computing Machinery and Intelligence, Alan Turing, 1950 (read via a course copy at redirect.cs.umbc.edu).
2. Large Language Models Pass the Turing Test, Cameron Jones and Benjamin Bergen, 2025 (preprint).
3. Minds, brains, and programs, John Searle, 1980.
4. The Chinese Room Argument, Stanford Encyclopedia of Philosophy.
5. Minds, Machines and Gödel, J. R. Lucas, 1961 (citation checked, not read).
6. The Emperor's New Mind, Roger Penrose, 1989 (catalogue record only).
7. Shadows of the Mind, Roger Penrose, 1994 (catalogue record only).
8. The Lucas-Penrose Argument about Gödel's Theorem, Jason Megill, Internet Encyclopedia of Philosophy.
9. Review of Shadows of the Mind, Hilary Putnam, 1995.
10. Minds, Machines, and Mathematics, David Chalmers, 1995 (Chalmers 1995b).
11. Penrose's Gödelian argument, Solomon Feferman, 1995 (some listings give 1996).
12. Gaps in Penrose's Toilings, Rick Grush and Patricia Smith Churchland, 1995.
13. Is Anyone Home? A Way to Find Out If AI Has Become Self-Aware, Susan Schneider and Edwin Turner, 2017.
14. Testing for Consciousness in Machines: an Update on the ACT, Susan Schneider, 2023.
15. Ex Machina's Scientific Advisor (interview with Murray Shanahan), Y Combinator.
16. Google engineer put on leave after saying AI chatbot has become sentient, The Guardian, June 2022.
17. Google fires software engineer who claims AI chatbot is sentient, The Guardian, July 2022.
18. On the Dangers of Stochastic Parrots, Emily Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell, 2021.
19. Role play with large language models, Murray Shanahan, Kyle McDonell, Laria Reynolds, 2023.
20. Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, Patrick Butlin, Robert Long and others, 2023.
21. Could a Large Language Model be Conscious?, David Chalmers, 2023.
22. Signs of introspection in large language models, Anthropic, 2025 (Anthropic 2025a).
23. The New York Declaration on Animal Consciousness, 2024.
24. The Edge of Sentience, Jonathan Birch, 2024.
25. Taking AI Welfare Seriously, Robert Long, Jeff Sebo and others, 2024.
26. Eleos AI Research.
27. Exploring model welfare, Anthropic, 2025 (Anthropic 2025b).
28. We must build AI for people; not to be a person, Mustafa Suleyman, 2025.
29. Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe?, Noemi Dreksler and others, 2025.
30. Folk psychological attributions of consciousness to large language models, Clara Colombatto and Stephen Fleming, 2024.

Appendix: who said what

Who's who

• René Descartes, philosopher. Mind and body as two substances.
• Thomas Nagel, philosopher. "What it is like" to be a bat.
• Frank Jackson, philosopher. Mary the colour scientist and qualia.
• Joseph Levine, philosopher. Named the explanatory gap.
• David Chalmers, philosopher. The hard problem, zombies, dancing qualia, and the 2023 paper on language models.
• Hilary Putnam, philosopher. Early machine functionalism, and later a critic of the Lucas and Penrose argument.
• Daniel Dennett, philosopher. Consciousness Explained and the deflationary view.
• Keith Frankish, philosopher. Illusionism.
• Galen Strawson, philosopher. The argument that physicalism leads to panpsychism.
• Philip Goff, philosopher. The popular case for panpsychism.
• Bernardo Kastrup, philosopher. Analytic idealism.
• Bernard Baars, cognitive scientist. Global workspace theory.
• Stanislas Dehaene, neuroscientist. The global neuronal workspace.
• David Rosenthal and Hakwan Lau. Higher-order theories.
• Victor Lamme, neuroscientist. Recurrent processing theory.
• Giulio Tononi and Christof Koch. Integrated information theory.
• Anil Seth, neuroscientist. Controlled hallucination, and biological naturalism about AI.
• Andy Clark and Karl Friston. Predictive processing and the free-energy principle.
• Michael Graziano, neuroscientist. Attention schema theory.
• Roger Penrose, physicist, with J. R. Lucas and Stuart Hameroff. The Emperor's New Mind (1989) and Shadows of the Mind (1994), the Gödel argument first made by the philosopher Lucas (1961), and Orch OR with the anaesthesiologist Hameroff.
• Francisco Varela and Evan Thompson. Enactivism.
• Alan Turing and John Searle. The imitation game and the Chinese Room.
• Susan Schneider and Murray Shanahan. The AI Consciousness Test, the Garland test (Shanahan's name for it) and role play.
• Emily Bender, linguist. Co-author of the stochastic parrots paper.
• Patrick Butlin, Robert Long, Jeff Sebo and Jonathan Birch. Indicator properties, AI welfare and the edge of sentience.

Summary table

How this was made: the idea, the practice and the final edit are mine. An AI assistant helped with research, fact-checking and copy-editing, and never invented experiences or quotes.