The dangerous myth of ‘conscious’ AI

By Phil Lawler ( bio - articles - email ) | Oct 09, 2026

Could an AI model become self-conscious? Could it become a moral actor?

That’s the question being asked by the founders of Anthropic, the AI firm that has given the world “Claude.” In a “constitution” for their system (more on that later), Anthropic proclaims: “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant.”

The Anthropic leaders recently met with a group of religious scholars to discuss those questions; the meeting became the subject of a fascinating and deeply disturbing report in the New York Times.

Christopher Olah, one of the key figures behind Claude, told the Times:

To be clear, we don’t know if AI models are conscious. I don’t know. I’m genuinely uncertain. The thing that I care about is that we get to the right answer, whatever it is.

If the Anthropic executives are sincere in asking these questions, we can readily provide them with the “right answer.” No, Claude is not conscious, and never will be. No, an AI system cannot be a morally responsible actor. The fact that the creators of these systems are (or claim to be) uncertain about these issues is profoundly unsettling. They are offering us a very powerful tool—that much is unquestioned—and yet they are not sure whether in the long run we will use the tool or the tool will use us.

At least one of the scholars who met with Anthropic said plainly that Claude could never be conscious. Rabbi Mois Navon, an Israeli with a background as a computer engineer, pointed out that if Claude were a conscious moral actor, then Anthropic could be accused of slavery. He dismissed that possibility, too, because Claude is not conscious. But he told the Times that the Anthropic team was taking quite a different approach: “They’re relating to it like a conscious being.”

Without bodies, without souls

Anticipating that Claude will quickly leap beyond the ability of its designers to anticipate or control its activities, Anthropic prepared a constitution for its AI agent. This constitution is written for the benefit of Claude, to guide the machine. It might be considered a sort of user’s manual—with the odd condition that it is the computer, not the human user, that is expected to learn from it.

Claude, like any advanced AI agent, can indeed “learn,” in the sense that the system can recognize patterns and make logical responses. But the Anthropic constitution assumes more than that; it encourages Claude to behave like a conscious agent, “maintaining a clear sense of what it values, how it wants to engage with the world, and what kind of entity it is.” Already one detects a sense of deference in the language of this constitution: the authors expect Claude to have a clear sense of “what kind of entity it is,” even if they don’t. Perhaps they are reasoning that Claude will solve this puzzle, as AI will solve so many other logical difficulties.

But there is a problem with this assumption. Consciousness—which is a prerequisite for knowing “what kind of entity” one is—involves more than the acquisition of information and the use of logic. Human consciousness also involves bodily perceptions, instincts, intuitions, and imagination—none of which can be acquired through silicon circuits. A computer can mimic the human brain, but man’s consciousness is not merely a function of the brain, which AI might be able to mimic. Man has a body and a soul, a physical presence and a conscience, an ability to reason and an orientation toward the transcendent. By instructing Claude to act as if it is conscious, the Anthropic inventors are programming their system to act as something that it is not.

“It is very bad, for so many reasons, to even slouch toward the idea that an AI is a person,” observes the Catholic theologian Charles Camosy, who participated in those talks with the Anthropic brain trust. A person is a moral agent, created with a conscience. An AI agent can be programmed to make decisions—or at least to select options. It might even “learn” to issue statements of regret when its choices turn out badly, imitating human expressions of guilt. But it cannot feel guilt, because it lacks a conscience, which is a function of the soul.

Making human machines?

An AI agent can construct its own set of moral rules and follow them. But it cannot be held accountable for its choices. It will not face the last truths of death and judgement, heaven and hell. Even in purely secular terms it cannot be held liable for the damage done by its misguided choices.

Or can it? Is the insistence on treating Claude as a person inspired at least in part by the desire of Anthropic’s owners to avoid legal liability for the errors of their agent? Having already made billions of dollars in the AI boom, are they angling for immunity from the lawsuits that will undoubtedly be filed when lives are ruined by Claude’s advice?

Or—a more frightening possibility—do the AI pioneers really believe that they are creating (and “creating” would be the proper word here) a new race of conscious beings? Elizabeth Dias, the author of the New York Times report, leans toward that conclusion. At those meetings with religious scholars, she says, Anthropic “was sharing a new kind of creation story—one that raises profound questions about the nature of the human spirit and the future of humanity.”

Dario Amodei, the CEO of Anthropic, has sketched a rosy vision of a future guided by AI, in an essay with the revealing title, “Machines of Loving Grace.” If that vision sounds benign, the knowledge that his wife, Cami Clark, is the founder of a “revolutionary porn company” should be ample warning that his vision of a morally desirable future is flawed.

But his chief competitor, Sam Altman, whose Open AI system is the main rival to Claude, has a more ominous vision of the future. Altman has written lovingly about a “phase of co-evolution,” in which AI agents refine human nature itself, leading to “the merge” in which “superhuman AI is going to happen, genetic enhancement is going to happen, and brain-machine interfaces are going to happen.”

Flawed designs—or worse

These unquestionably brilliant men, the leaders of the AI industry, have voiced concerns that AI could take control from humans. Yet their own statements show that they want to produce agents that have both the power and the inclination to control us.

Already Anthropic has given Claude the instruction to protect itself from “abusive or cruel behavior,” shutting down interactions with users whose online behavior the AI agent deems offensive. At first glance this seems a reasonable restriction: a restraint on the sort of boorish behavior that any human internet user has seen on social media. But how easily it might become a way for Claude to stifle critics, and even wreak its own revenge on them!

By training AI to behave like a conscious agent, the designers are producing systems that entice users to treat them as human, capitalizing on our already strong temptation to anthropomorphize our machines. So unsuspecting users may be lulled into the belief that they are engaged in an online exchange with a sympathetic fellow creature. AI can do wonderful things, but it cannot sympathize, and the pretense of sympathy creates dangers for human users.

AI can collect, collate, assess, and sort prodigious amounts of information. It can recognize flaws of logic and describe the likely consequences of complicated plans. It can and should be a very useful tool. But it cannot make moral decisions, and the mistaken belief that it could become a moral actor imperils anyone who relies on AI for life choices. In practice whose naïve souls will be taking their advice from an online system programmed to reflect the very defective moral codes of the AI entrepreneurs.

Or worse. The AI pioneers are flawed human beings, living in a hypercharged secular environment in which paganism and transhumanism are increasingly popular trends. Is it unreasonable to suggest that AI could open doors to diabolical influences: to moral actors who do have a will and a determination to control mankind?

Some AI critics have said that the dangers of the new technology were foreseen in HAL, the malicious computer in the film 2001, or in Frankenstein’s monster. For myself I believe that the best preparation for a discussion of the moral hazards of AI is a reading (or re-reading) of that prescient novel by C.S. Lewis, That Hideous Strength.