No Consciousness? No AGI!

Why (unconscious) AI can't attain human-like intelligence

A post-material worldview isn’t just an ideal or abstract spiritual philosophy; it is intended to change the world for the better, including from a pragmatic and material standpoint. When we begin to see beyond the strictly materialistic and mechanistic worldview that still strongly dominates our culture, this shift will also have very concrete and practical consequences. To illustrate this, one can take the reverse approach by showing how a lack of deeper insight into our own inner nature can blind us, luring us into false belief systems, and potentially lead to the waste of enormous resources. The specific case study I often like to address in this regard is the modern, all-pervasive discussion on the coming AI or even AGI (Artificial General Intelligence) revolution that is purported to shape our future.

After the success of ChatGPT, and due to the impressive capabilities of large language models (LLMs), there has been a noticeable surge in discussions and articles about the imminent arrival of AGI—that is, human-like intelligence. We have seen various breakthroughs and advancements, and some experts are even predicting AGI within the next few years. After all, it seems reasonable to believe that if machines have become so smart as to pass medical, law, and business school exams, it must be only a matter of time, perhaps only a few years, before we will see the first humanoid intelligence talking to us, possibly working for us, causing mass unemployment, eventually becoming much more intelligent than us and, in the worst-case scenario, taking control and dominating the human race like in a Terminator movie doom-and-gloom scenario.

Let me explain why this is nonsense.

These much too optimistic beliefs—or pessimistic, fear-based projections—are not primarily the result of flawed reasoning or a lack of technical understanding of what intelligence is. Instead, they stem from a psychological inability to truly know ourselves and from an overly externalized mindset that has forgotten how to adopt a first-person perspective and view things from within.

In a previous post, I explained why, despite its remarkable capabilities, modern AI doesn’t understand a thing. This may sound like an overly negative assessment of what large language models (LLMs) can do. When we interact with a chatbot, it can give the impression that it genuinely understands what it's doing. In a certain sense, we might even say that modern computers have a good grasp of the data they process, particularly with language. However, it's more accurate to describe this as the appearance and imitation of deep understanding, rather than what I would call 'semantic awareness' in the machine.

I provided several examples that demonstrated not just how AI makes mistakes—something we are all aware of, as humans make mistakes too, which is not the point—but how the absurdity of some answers reveals its lack of deeper understanding of what it’s doing. Nevertheless, the prevailing belief was that achieving AGI required simply feeding these systems with more data, more computing power, more neurons, trillions of parameters, and more of the same.

However, scientists are gradually realizing that it isn’t that simple and that there may be fundamental flaws in our understanding of what intelligence and reasoning are about. They are beginning to sense that the gap between syntax and semantics is likely much wider than previously thought. Even with enormous data sets and immense computing power, a computer will not gain a true understanding of the real world.

For those with a technical background and who have some time to explore where we currently stand and what the real challenges are, I recommend the lectures by François Cholletere and Gary Markus. For a reflection on what 'artificial general intelligence' could truly mean, consider reading Melanie Mitchell’s article. If you're interested in a more humanistic and less machine-centered perspective, Shannon Vallor’s article comes closer to the point I’m trying to make: “We are more than efficient mathematical optimizers and probable next-token generators.” It seems that most AI/machine learning researchers aren’t willing (or aren’t able?) to even consider this eventuality. Moreover, in a not-so-different context, noteworthy is also how, despite all the investments and the hype, there will be no fully autonomous self-driving cars anytime soon.

Overall, what researchers are beginning to realize is that despite billions being poured into R&D, LLMs continue to hallucinate, fabricate information, lack any true representation of the world, excel in deep learning but falter in generalized abstract reasoning, struggle to understand wholes in terms of their parts, and frequently fail on tasks involving object and spatial continuity. It’s becoming clear that, in terms of AGI-level performance, there has been no substantial progress: there is still no true reasoning, no common-sense knowledge, no 'ghost in the machine' with semantic awareness, and certainly no independent agency. Deep down, what LLMs do is next-word predictions: given a string of text, they calculate the probability that various words can come next. Ultimately, these systems remain passive mechanisms that rely predominantly on pattern recognition and best-guessing. We are still far from developing systems with true reasoning and the current hype feels reminiscent of past AI bubbles, where inflated expectations outpace actual capabilities.

To make a long story short, the issue boils down to recognizing a fundamental cognitive gap between symbols and the meanings they convey. Data and the relationships between data are insufficient on their own. True understanding requires more than that.

Interestingly, this isn’t a new insight. It was first highlighted in the 1980s by John Searle’s famous ‘Chinese room’ thought experiment and further emphasized by the ‘symbol grounding problem,’ discussed by Stevan Harnad in 1990. To summarize, the symbol grounding problem, points out the persistent challenge of determining where symbols (such as words, numbers, streams of bits, or signals) derive their meaning. There is a gap between symbols and signals ‘out there’ and our semantic awareness, intuition, understanding, and knowing of their meaning. There is something that remains outside the scope of formal computational models. More recently, in 2019, Froese and Taguchi revisited the issue in their discussion on “The Problem of Meaning in AI and Robotics: Still with Us after All These Years.”

There is an elephant in the room that only a few are willing to acknowledge: the possibility that human-like intelligence, with its semantic awareness, may not be naturalizable and might never be fully replicated in a machine (unless it becomes conscious.)

It isn’t necessary to invoke sophisticated metaphysical speculations that posit a duality between mind and body. There is a very simple experiential fact that nobody denies, but almost everybody ignores. If we take a first-person perspective, it is easy to realize how our semantics-based cognition is directly related to a conscious experience. Always.

You cannot truly understand colors, sounds, tastes, smells, temperature, touch, or sight if you have never had a conscious experience of seeing a color, hearing a sound, tasting chocolate, smelling an odor, or touching an object. One might also tell you everything about the H2O molecule, chemistry, and physics, but you cannot comprehend what wetness truly means unless you have experienced the wetness of water yourself.¹ Without a subject—an entity or someone—to have a conscious qualitative sensation; so called ‘qualia’—of these things, you could provide all the textual descriptions imaginable but the machine will never grasp anything beyond the purely abstract apprehension of symbols, letters, words, sentences, and numbers.

No matter how advanced or sophisticated an information processing system may be, it cannot understand what an image of a street, a cyclist, or a traffic light represents. Of course, the image can be converted into vector spaces, matrixes, relationships, curve fittings, probability laws, and who knows what. But, unless the same light signals on the pixels forming the image on the CCD camera are converted into a subjective lived experience, all these mathematical abstractions will remain meaningless entities. To truly comprehend these things, one must have physically experienced the environment. For example, you need to have felt the weight of your body walking on that street and have had conscious, experiential interactions with others through sound, speech, and vision. You cannot drive a car without a semantic understanding of the environment—the street, the cyclist, the traffic lights, and so on. There's no reason to believe that a self-driving car could magically understand what even humans cannot grasp without conscious experience.

Thus, there is a tight and inextricable relationship between feeling and knowing, sensing and understanding, perceiving and comprehending. You can’t disconnect these two aspects of cognition and treat them separately, as current research is doing.

In short: You cannot understand anything without consciousness—not even in principle.

The same applies to any AGI narrative. There is a direct relationship between general intelligence and consciousness. AGI cannot exist, even in principle, without becoming conscious, because true intelligence requires semantic understanding, and semantics is rooted in sentient experience. Adding another trillion neurons, an astronomical number of parameters, flooding an AI system with more data, or increasing its number-crunching power won’t do the trick. Without consciousness, there is no AGI. After all, from a first-person perspective, it becomes self-evident.

The fact that, sooner or later, the machine makes an extremely stupid mistake that reveals its lack of understanding and, thereby, shows that it has no subjective experience, has also other implications. For example, this constitutes a strong argument against the hypothetical existence of so-called "philosophical zombies," or "p-zombies" for short. A p-zombie is a hypothetical being imagined to be physically identical to a normal human but lacking an inner life and conscious experiences, or qualia, yet behaving exactly like a conscious human being. However, since a p-zombie would lack the sentience necessary for meaning-making, it would eventually make extremely stupid mistakes, just like other non-conscious AI systems, thus betraying its existence. It is reasonable, therefore, to conclude that beings that act like humans but lack subjective experiences do not exist, or we would have already been able to distinguish them from conscious humans. In my opinion, a meaning-making test is a much stronger and potentially much more convincing proof of the absence or presence of consciousness and intelligence in a computer than the Turing test.

Thus, if there will be any AGI, then we will have to build conscious machines first. And science is still light years away from even beginning to understand how consciousness emerges in organic intelligence itself, and how it relates to agency and life.

Therefore, there is a foundational and principled argument against the coming of an AGI. Unfortunately, IT scientists often dismiss philosophical arguments and need to learn these lessons the hard way. They are only now beginning to confront what was already understood four decades ago.²

The root cause of this over-trustfulness in an AGI revolution is that humans don't truly know themselves. We are continuously externalizing our focus and are too prone to objectifying reality, which has made us unable to see, perceive, or feel how our own cognition works. If we hadn't insisted on a mechanistic vision of the mind and life, we might have been less naive and could have directed our investments toward more fruitful directions, and, as a bonus, become more aware of what consciousness, cognition, and life truly are.

In cognitive sciences and biology, scientists are confronted with exactly the same problem, only coming from the opposite direction. Consciousness, mind, life, cognition, and agency are already there, and we wonder how this could possibly be. Now there is a slight shift towards a less reductionist worldview that, however, is still firmly anchored in a materialistic understanding of reality. Some try to 'inject' free will into life but aren't willing to give up a strictly deterministic view of reality (the so-called 'compatibilists', as I already discussed here). Others are willing to give up the machine metaphor in biology since it is now clear that it isn't working, but insist on preserving the blind mechanistic processes as the fundamental primitive of life. While, the theory of enactivism understands that the old naturalism is gone and wants to introduce a "new naturalism," but realizes that it still can’t account for the emergence of experience in the natural world. Ultimately, we persist in trying out more of the same and would like to keep one foot in two shoes and have it both ways.

More academics are recognizing that something is missing in our understanding of what it means to truly 'understand.' They point out that language alone can’t lead us to human- or animal-like knowledge; context is also needed. However, context alone isn’t enough either. The next step is recognizing how consciousness plays a fundamental role in making the transition from symbols to semantics.

We are not there yet, but I believe the time will come when we realize that the failure to build an AGI will reveal something about ourselves, albeit in reverse. It won't explain how our mind works or how consciousness emerges, but it may show us what they are not and where they do not originate. It may demonstrate that consciousness, mind, and life are not mere epiphenomena of complex material machinery but are something more fundamental. Once this understanding becomes common wisdom, we will collectively transcend the straitjacket of a mechanistic and material conception of reality, particularly regarding life and ourselves. Once this shift is established, the world may truly change for the better. And, maybe, will rediscover this ancient wisdom.

The subscription to Letters for a Post-Material Future is free. However, if you find value in my project and wish to support it, you can make a small financial contribution by buying me one or more coffees!

1. This doesn’t imply that, for example, congenitally blind people can’t understand others talking about light, colors, or other visual sensations. Of course, they do. However, they gain only an indirect understanding of it related to the information they gain from the experience of these qualitative phenomena through other people, or from other senses. The “seeing” goes by sound, touch, taste, and smell. This allows them to semantically represent and comprehend visual concepts, even if they can't directly perceive them, precisely because the representation is backed by other forms of conscious experience.

2. In principle, this could even suggest a new test for intelligence, potentially replacing the infamous Turing test. The idea would be to design a test that reveals whether a machine possesses real semantic and deep understanding, akin to that of a human. If it does, then this would indicate a form of 'intelligence' that suggests the presence of experience as well.