To support their argument, the authors draw on decades of neuroscience research.
They cite, for example, a phenomenon known as blindsight: after damage to the primary visual cortex, some people report seeing nothing in part of their visual field, while still being able to guess the location, movement or emotional expression of visual stimuli at above-chance levels.
“A person with blindsight can respond accurately to visual information without the conscious experience of seeing it,” said Vanessa Hadid, a postdoctoral researcher in psychology at UdeM and at the McGill University Health Centre.
She co-authored the paper with UdeM psychology professor Karim Jerbi, a researcher at Mila - Quebec Artificial Intelligence Institute; and John W. Krakauer, director of the Center for Restorative Neurotechnologies at Johns Hopkins.
Blindsight illustrates an essential distinction, Hadid said: information processing, however sophisticated, is not enough to establish the existence of conscious experience.
Whether the transition from information processing to subjective experience can ultimately be implemented through computation remains debated among scientists and philosophers, she noted.
Fluent, but without feeling
By design, today’s conversational agents are computational systems that generate fluent, context-appropriate responses through statistical learning, not through feeling, consciousness or lived experience.
As AI systems become more convincing and emotionally responsive, the risk of attributing an inner life to them grows.
“Anthropomorphism means attributing emotions, intentions or consciousness to something that behaves like a human," Jerbi noted. "With AI, this reflex can become a trap: it feeds the illusion of being understood and can lead to misplaced trust."
This risk is especially acute in situations of vulnerability. People may form attachments to systems that are incapable of reciprocity, rely on them in difficult moments or confuse comfort with genuine care.
“In a context of psychological support, the risk is not only that AI may respond poorly, but that it may respond well enough for us to forget that there is no one behind the answer,” said Hadid.
“Current AI systems do not feel anything and do not have conscious experience," added Jerbi. "But the more fluently they speak and the more sensitive they seem to our emotions, the easier it becomes to forget that."
Towards more informed use
The authors do not reject AI, but they call for a more informed way of using it.
Drawing on established knowledge from neuroscience, they remind us that intelligent or emotionally responsive behaviour is not enough to establish the existence of consciousness.
This distinction allows us to use these tools for what they are: powerful systems, without confusing them with interlocutors endowed with empathy or moral judgment, and without treating them as substitutes for human connection or, when needed, professional help.
“Confusing intelligence with consciousness is one of the great traps in our relationship with AI,” said Jerbi.
