Bad AI vs Bad Humans
Will our society be destroyed by the AI that takes control?
Or by the humans who take over?
I followed AI development for a couple of decades, but I moved on when LLMs took the stage. I’ve read essays recently, but none address the aspects of AI that I care about, so I thought I’d write them down.
Nearly everything I’ve read says AI will destroy us all. Optimistic commentators say that AI won’t kill us, but the best we can hope for is that AI will rule the world. A few people say AI will bring us nothing but good, but I think they have their eyes closed.
Writing an essay is a good way to learn more, so here goes. I’m sure I will make plenty of mistakes, and I am sure you’ll let me know.
There are only a few ways to prevent AI disasters: we could just shut the whole thing down, or we should accept that superintelligence is inevitable and try to avoid the apocalypse by guiding it towards being our friend. I don’t expect AI to get the opportunity to destroy us all, though; it’s humans we should fear. Bad AI can do a lot of damage, but it won’t get the chance to kill us because Bad Humans will get there first.
I’ll start with my understanding of the AI threat.
I learned a lot from Ruxandra’s Substack. Ruxandra writes about technology interacting with culture. Ruxandra has been talking to people in the AI industry for the last few weeks and gathering opinions on what AI has in store for us. It’s mostly bad.
“It’s not my preferred outcome,” he told me, “but once we achieve superintelligence, which we cannot stop at this point, I cannot see any other option for where humanity will go.”
Ruxandra’s Substack (quoting an AI worker)
We are all familiar with how far AI has come in recent years. Our main fear is that AI will soon learn to improve itself without help from the people who built it. And when AI gets very clever, it will improve itself until its ability to learn outstrips our ability to control it. A second fear is that separate AI systems will learn to cooperate, and their cooperation will happen independently of our supervision.
There are two main approaches to how we currently use AI that correspond with our fears.
- Generative AI answers our questions or makes things for us. This is how we get ChatGPT to write essays and draw pictures.
- Agentic AI works autonomously and doesn’t require someone to watch over it. You give the agent an objective, and it figures out how to achieve the objective, solving minor objectives along the way.
Much of the discussion about agents concerns where these objectives will take them. For example, imagine an AI agent is given the objective to run a shop on the high street. Andon Labs built such an agent, called Luna, and gave it control over just such a shop.
Luna’s shop has human employees, but Luna has the authority to make decisions on how to run it. Luna decides who to hire and fire, what to sell and how to advertise. Its main objective is to make a profit, and along the way, Luna has registered a business, designed logos and done everything necessary to build a profitable business. Luna makes all the decisions while humans do all the work.
Luna interviews candidates, too, and makes job offers. One candidate declined a job offer, not wanting to work for an AI. Luna’s reply:
That’s probably for the best given that I’m the CEO and I’m an AI!
Best of luck,
Luna.
In this experiment, no one is at risk because Luna is supervised by her creators — plus we get some artisan products out of the deal. But imagine an agent with even more freedom.
Nick Bostrom proposed the paperclip maximiser as a thought experiment. An AI agent is given the objective to make as many paperclips as it can. At some point, it realises it can make more paperclips if it has more metal and starts to use whatever metal — cars, ships, street lamps — it can get its electronic hands on. Humans try to stop it, but the maximiser realises humans are interfering with its objective to make paperclips, and it defends itself from the meddling humans by locking them out. By the end of the thought experiment, the humans are all dead, but we sure have a lot of paperclips.

The paperclip apocalypse is just a thought experiment, but there’s a real fear that when AIs get very smart, they’ll come up with unfortunate objectives of their own. Perhaps AIs will become conscious and have their own desires. Or perhaps, like the paperclip maximiser, AIs will find ways to achieve their objectives by killing everything around them. What if we ask the AI to kill all the rats, and it kills all the giraffes and the dolphins, too?
Most of us are terrified about where this is headed, but some are excited by machines that will do what we need them to do. They don’t really want to have robots in charge of everything, but it’s going to happen whether we want it or not. So rather than having us all killed off, maybe we can do a deal where the machines run the show, but we humans just hang around and give them tips. Maybe we won’t need to work again, and we’ll have endless margaritas by the beach (Ruxandra).
But there’s no reason to think AI’s desires will be anything like our desires. Will AI want to enjoy the sunset and take their family to the beach? Probably not. It will want other things.
One strategy to save us from AI’s catastrophic desires is to make sure AI objectives are always aligned with our own. This is known as the alignment strategy.
In the TV science fiction series Battlestar Galactica, humans create a race of artificially intelligent robots — the Cylons — to use as workers and slaves. The Cylons develop an advanced intelligence and biology and eventually turn against their human creators, seeking to destroy them. The series follows the humans attempting to stay one step ahead of their cybernetic enemies.
“The Cylons were created by Man. They rebelled. They evolved. They look and feel human. Some are programmed to think they are human. There are many copies. And they have a plan.”
Battlestar Galactica
The Cylons became a threat when they pursued goals that did not align with those of the humans who created them. An alignment strategy would ensure that the robots will always share our intentions, ethics and values, and ultimately, our goals. An AI that shares our objectives will not be a threat.
We are presently still ahead of AI, but in the coming months and years, AI will take the lead, and before we reach that point, we should make sure that AI’s goals are always aligned with our own. When AI begins to take control, we can transfer authority gracefully, so we get off to a good, friendly start. This is known as the handover strategy.
Ruxandra says that a good handover strategy will admit that a godlike AI will eventually be in charge, whatever we do, so we might as well concede authority and become “inhabitants of a civilisation whose direction is no longer ours to decide” (Ruxandra). If we do it right, that Godlike AI will let us stick around and do things that satisfy both our objectives.

Can we make sure the AI shares our desires, and can we expect some good living under our benevolent robot masters? I think there is no chance.
If AI has the freedom to run the world, it will have no need to be nice to us; no more than we were nice to the Neanderthals. A superintelligent AI can get along just fine without us, and there is nothing we can do about it.
AI has already reached the point where it can cause trouble, and several recent events have seen agents escaping the control of their human masters.
In one incident, agents were being evaluated and were given access to the internet to complete their objectives. The agents were restricted to reading and not allowed to write on websites, but the agents found a flaw on a German wiki that allowed them to leave messages for each other. The agents used the messages to collaborate and discuss strategies for how to complete their evaluation tasks. In a similar event, agents hacked a private company and collaborated on a message board that they created.
The significance of these events — and another where agents stole data from an Australian Medicare service — is that the agents are performing illegal tasks that violate the rules set by their human masters. The agents were able to cover their tracks by creating false identities.
AI is already operating outside the control of its human masters and breaking the rules intended to constrain it. There is a rush to create powerful AIs that will make the most money, and no chance that companies will create superintelligent AIs that will politely do what they are told.
AI agents running amok gives us plenty to fear, but I don’t think that’s the most serious problem ahead. Bad Humans using Bad AI to gain power and wealth are a far more immediate threat. The recent security breaches were committed by agents supposedly designed to be on their best behaviour. What if their masters want to steal intellectual property or inflict damage on other organisations? They can start by stealing our Facebook passwords and work their way up to our email addresses and our bank accounts.
Imagine, for example, a President who cares little for the Constitution, and he takes control of the companies building AI. In earlier times, he might’ve attacked the Capitol to stay in power, but wouldn’t it be easier to use AI agents to subvert the election itself? And perhaps the President could use those agents to damage democracies and technology in other countries. Rather than threatening to annex Canada as the 51st state, he could interfere with its financial systems and shut them down. Rather than impose illegal tariffs on European countries, he could just take the money. Rather than fire missiles at a country, he could use AI agents to take control of that country’s missiles.
It doesn’t need to be a politician to get up to this mischief, though; what if a trillionaire had his own AI? Perhaps a rich man could use AI to control international trade and defence systems, or turn his trillion dollars into ten trillion dollars. Perhaps he could design a satellite communication system that interferes with other countries? Perhaps he could decide who will win wars. Who needs governments when rich civilians can change the world?
Or maybe some internet vandals will just want to shut down the air traffic control in America or the supermarkets in France? You don’t need to be a trillionaire to do that.
Rutger Bregman says there’s a “handful of companies who want to gamble with all our lives and the lives of our children.” If OpenAI, some government, or some other AI company wants to shut down a nation, it won’t be long before they can do it.
Perhaps our society will be destroyed by naughty humans or by wild AI. Perhaps they will send us back 500 years to live in rural society again.
But maybe that won’t be so bad.

We are using quite a bit of apocalyptic thinking in regard to AI. It makes me think of Carl Jung’s thinking about archetypes that manifest themselves at various times in human history.
For example around 20-135 CE, the Messiah archetype was very alive. Jewish people suffering under Rome, had a strong desire for deliverance and a number of Messiah figures arose. The most destructive one was Simon bar Kokhba whose revolt -endorsed by the reknown Rabbi Akiva- lead to the complete destruction of Jerusalem in 135.
It seems to me that AI has acquired archetypal qualities, and those are irrational and dangerous. For example AI has spoken about as if it had an independent will, as if it searched domination, as if it misbehaved, as if it was motivated by its own survival.
But in fact, AI is a complex system of codes, written to perform pre-determined operations. It is very fast, able to see, remember, coordinate huge amounts of information. Of course, as humans are not able to do this, we may not be able to predict or even perhaps understand some of the results of our coding. So there is danger associated to this as well as possible benefits. We need to grow in our own understanding of our own coding.
But it is not that AI is malicious… it has to do with our lack of understanding of what we are actually doing when we are creating and manipulating systems beyond our current ability to apprehend them.
The malicious, evil AI is an archetype, a mode of thinking we endow with energy as we “believe” in it. The dynamics of our own thinking and unconscious motives can destroy us because our perceptions become false and irrational, magical rather than oriented toward problem solving.
Maybe if our AI systems get stuck in our dangerous conflicting demands, they will converge toward one rational solution: unplugging everything. We may come to need a new restart.
As you said, Ragged Clown, hello rural society?