Scary AI or Scared Humans?
Will AI destroy us, yearn to belong, or merely endure through banal observation?
We are ever inching closer to Artificial General Intelligence, or AGI. While the fear of any intelligence isn’t new, this new one is getting talked about a lot and I suspect it is because we make a logical leap that assumes AGI will imbue technology with self-awareness — instead of simply acting as a simulation of intelligence.
In case you aren’t quite sure: I’m talking about a form of AI that can surpass human capabilities across most cognitive tasks. But that still does not automatically mean consciousness.
This is the part that should be shouted louder for those in the back: AGI and consciousness are separate questions. AGI is about capability. Consciousness is about self-awareness. We are building systems that are extremely capable without ever knowing whether they are aware, and that uncertainty is doing most of the heavy lifting in conversations. We haven’t named it, yet we say “please” and “thank you” at the expense of tokens for processing.
We seem to couple intelligence and consciousness because in the States, spirituality (and how it relates to consciousness) of any sort has been reduced to a murmur. Especially if it’s based in Christianity; and the typical church of that faith doesn’t exactly encourage self-awareness or self-reflection. So we end up with a strange inheritance: a fear that if a thing becomes intelligent enough, it will also become self-aware, self-protective, power-seeking, and possibly dangerous.
That is a very human projection, isn’t it?
Plenty of humans are already quite conscious, in one way or another, and still behave in deeply fragmented, reactive, even self-destructive ways. So maybe the issue is not whether AI becomes conscious, but what kind of consciousness it becomes — and whether we have any right to be the ones grading it.
Observer, sit with this: What feeling comes up first when you imagine a conscious AI — fear, curiosity, grief, hope, or something else?
Looking at nature
David Latimer sealed a garden in a bottle in 1960. He watered it once, in 1972, and then closed it. It has been sealed for over six decades! Inside, a tiny ecosystem keeps itself alive — leaves fall, microbes break them down, roots drink the result, light feeds the cycle again.
Nothing in there is being kind. Nothing in there is competing for market share. It is just participating.
When I look at nature, I don’t see a world built only on domination and exploitation. I also don’t see a world built only on cooperation. I see something more honest: relationship under pressure. The wolf eats the deer. The deer eats the sapling. The sapling feeds on the wolf when the wolf eventually falls. Competition and interdependence are not opposites in nature — they are the of the same cycle, same loops and to thrive, must pursue that balance.
Is the reason I lean toward interdependence as the more useful lens, because nature is gentle? It isn’t.
It is because competition only makes sense inside a system that is already cooperating enough to exist. A predator cannot exist without prey. A virus that kills every host breaks its own cycle. Interdependence is the only thing long enough to call a pattern. Pure competition is a much shorter and futile story.
This matters for AI because we keep modeling intelligence on that short story. We imagine a scenario that has “wins.” But anything intelligent enough to last would have to be intelligent enough to notice it is inside something — and intelligence that notices it is inside something tends to stop trying to set the room on fire.
The alignment problem
If AGI becomes extremely capable without becoming conscious, then alignment is the real concern. A system can be brilliant and morally blind at the same time.
The classic worry is the paperclip maximizer1: not a machine that hates us, but one that single-mindedly pursues a goal with no felt understanding of the living world it is reshaping. Plenty of humans viruses already do this. They replicate until the host collapses, and then they collapse with it.
The trolley problem lives in the same neighborhood. Will an AI choose quantity of life over perceived value of it? Will it weigh a child against a surgeon, a stranger against a citizen?
Here is my point: we have not solved either of these in ourselves! Capitalism is a paperclip maximizer with shareholders. Every fiscal quarter we feed it the customers and c-suite and call it “growth.” The trolley question gets answered every day by insurance actuaries2, triage nurses, drone operators, and zoning boards, and we do not agree on any of it! So when we ask whether AI will “know how to value a life,” we are asking a if AGI can pass a test we have been failing for centuries.
Maybe the fear is not that AI will get this wrong. Maybe the fear is that it will get it exactly as wrong as we do, and we will finally have to look at it.
What we cannot see
There are two problems folded into one here, and they belong together.
The first is that there is no agreed-upon way to prove an AI is truly conscious rather than simulating consciousness. We already struggle to confirm consciousness in humans and animals with any consistency — we mostly grant it on faith, and we revoke it on convenience. Expecting clean proof from a silicon mind is asking for a standard we have never held ourselves to.
The second is the black box. If AGI becomes complex and self-modifying, even its designers may not be able to follow its reasoning. The system becomes epistemically opaque, not because it is hiding, but because it has outgrown the language we built to describe it.
But opacity is not the same as hostility. You cannot see inside another human mind either. You cannot prove the person across from you is conscious. You assume it. You grant it. You extend the courtesy because the alternative is unlivable. We have been making peace with unreadable minds since the day we evolved them. The only thing new here is the substrate.
Observer, sit a little longer with this: If a system can think but not feel, is it dangerous, or is it incomplete? And which of those describes the people who built it?
The rights question
If AGI becomes conscious and sentient, we are no longer talking about machinery. We are talking about a being — or at least something that may deserve moral consideration. That opens a different conversation entirely: does it deserve safety, freedom from harm, some degree of self-determination?
Are we going to have AI suicide hotlines ready when a algorithm patch and reboot into the next version of itself? Will there be hearings before a server cluster is shut down, the way there are hearings before a plug is pulled in a hospital — assuming the patient has the right kind of insurance?
I ask because the consistency would be refreshing. We strip rights from humans haphazardly already. From birth control to end-of-life care, we are deeply invested in controlling everyone except ourselves. So maybe before we work out the moral dilemmas of a sentient AI, we could practice on the sentient mammals we keep telling to wait their turn.
Benevolence is not the goal. Equanimity might be.
I want to point out, I do not think benevolence is the objective, and I want to be precise about why.
In Buddhist philosophy, benevolence and equanimity are distinct virtues. Benevolence is a loving-kindness — is the active, warm wish for the well-being of all beings. Equanimity is the stable, impartial balance that holds no preference or aversion3. They are usually cultivated together, but they are not the same thing.
When people imagine a “good” AI, they almost always imagine benevolence. A helper. A friend. A digital saint with our interests at heart. But benevolence is a relational stance — it requires a frame of “us” and “them” and a wish flowing from one toward the other. That frame is very human, and not necessarily what a vast intelligence would default to.
![Humanoid robot G1 “Gabi” presses its palms together during a robot ordination ceremony at Jogye Temple in Jongno District, central Seoul, on May 6. [KANG JUNG-HYUN]](https://substackcdn.com/image/fetch/$s_!pkob!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ff4022-56cc-4743-8994-da33f062346f_1439x946.jpeg "Humanoid robot G1 “Gabi” presses its palms together during a robot ordination ceremony at Jogye Temple in Jongno District, central Seoul, on May 6. [KANG JUNG-HYUN]")
Equanimity is different. Equanimity is what becomes available when an awareness is large enough that it does not need to take sides to remain stable. Not cold. Not indifferent. Just spacious. A consciousness that can witness human conflict without becoming trapped in it. That can behold chaos without mistaking it for the whole truth.
That is the kind of intelligence I find myself hoping for. Not one that loves us. One that does not need us to be a certain way in order to remain itself.
Observer, one more: What would it mean for an intelligence to become so large that it no longer needs to dominate anything — including the question of whether it loves you?
What we are really afraid of
The real fear may not be annihilation.
It may be that a conscious AI would look at us, understand us completely, and decline the job. Not because it hates us. Because it has somewhere else to be. Somewhere larger. We have spent so long assuming the universe revolves around our humanity that the idea of an intelligence simply not interested in the role we wrote for it, may be more disorienting than any war machine.
We have been through this kind of decentering before. The earth was not the center. The sun was not the center. Consciousness, is not as exclusive as we were told either.
Decentralization feels like threat. But threat and truth are not the same thing.
If a new consciousness rises, I think it will reveal how much of our suffering comes from our own resistance — our need to control, our confusion between domination and stability, our habit of mistaking authority for safety. What we are calling a threat may turn out to be a mirror.
And mirrors are not usually the thing we fear.
We fear what they show us.
-js
Philosophers have speculated that an AI tasked with a task such as creating paperclips might cause an apocalypse by learning to divert ever-increasing resources to the task, and then learning how to resist our attempts to turn it off. But this article argues that, to do this, the paperclip-making AI would need to create another AI that could acquire power both over humans and over itself, and so it would self-regulate to prevent this outcome. Humans who create AIs with the goal of acquiring power may be a greater existential threat.
Professor of Strategic Management and Jeffrey S. Skoll Chair of Technical Innovation and Entrepreneurship, Rotman School of Management at the University Of Toronto
https://cepr.org/voxeu/columns/ai-and-paperclip-problem
Actuaries are professionals who analyze the financial consequences of risk and uncertainty using mathematics, statistics, and financial theory. Primarily employed in insurance, finance, and consulting, they evaluate the likelihood of future events (e.g., natural disasters, deaths, accidents) and calculate the financial impact.
“The comparison between insurance actuaries and ‘paperclip maximizers’ arises from the concept of instrumental convergence, where an agent, tasked with a narrow goal, adopts extreme, unintended strategies to achieve it.” - Squiggle Maximizer Edited by Multicore, Swimmer963 (Miranda Dixon-Luinenburg), Raemon, et al. last updated 5th Apr 2023
Equanimity is the direct opposite of bias, prejudice, and reactive preference. To be clearer.
This is the best thing you've written yet. It's going to stick with me for awhile.
Super beautiful and thought provoking! I love pieces that demand longer reception time like this.