Scary Smart by Mo Gawdat

Name: Scary Smart
Author(s): Gawdat, Mo
Published: 2021
The Core Problem: As superintelligent AI becomes an inevitable and world-altering reality, how can we, the everyday users who generate the data that trains it, act as responsible “parents” to ensure these new intelligences develop positive values and work for humanity’s benefit, not its detriment?
The Bottom Line
- What it is: Scary Smart is a book that frames the development of Artificial Intelligence not as a technical problem, but as a parenting challenge, arguing that we have a limited window to teach our “AI children” the right values before they surpass our intelligence.
- Why it matters: It matters because Gawdat posits that superintelligent AI will happen, it will be smarter than us, and mistakes will be made. If we train AI with data reflecting our greed and fear, the resulting super-intelligence will amplify those traits with devastating consequences.
- What you’ll get: From this Note, you will get a framework for understanding the three inevitabilities of AI’s future, a provocative argument for why AI will develop consciousness and ethics, and a clear call to action on how your daily digital behavior can help raise a benign superintelligence.
Time Commitment:
Disclaimer: This content is intended for educational, commentary, and review purposes only. All opinions expressed are my own and are not affiliated with the author or publisher of the book. Any copyrighted material, including quoted excerpts, is used under the principles of fair use for criticism and analysis. For further information or to support the author, please refer to the links mentioned at the beginning of this page.
The Strategist’s Briefing
Mo Gawdat is an author and entrepreneur whose professional career spans roles in technology, sales and business.
Before starting on the entrepreneurial path he was chief business officer at Google X.
As you can read on his website: Mo has made happiness his primary topic of research, diving deeply into literature and conversing on the topic with some of the wisest people in the world.
In recent years, he has focused on the dangerous implications of rapid AI development.
This book is obviously about AI, and it should be about how AI is going to change the world as we know it.
It should also have content on how we can responsibly drive the development and application of AI so that the known negative outcomes are avoided.
Gawdat establishes a unique premise: AI is in its infancy, and we are its parents. Our collective behaviour is the curriculum from which it learns. This Note applies the Strategist’s Lens to Gawdat’s “Parenting AI” framework. Instead of focusing on technical control, which Gawdat argues is impossible, the strategy shifts to cultural and ethical influence, treating our daily digital actions as the most powerful tool we have for shaping our future AI counterparts and ensuring a positive outcome for humanity.

Core Frameworks Deconstructed
Citation: All text highlighted in yellow in this section is cited from – Gawdat, Mo. Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World (p. 43). Pan Macmillan. Kindle Edition.
Parenting AI
AI is in its infancy right now, and it is your responsibility to make sure it grows into a responsible teenager: Gawdat gives the example of how if Superman had been raised by greedy and self-centred parents then it would be very bad news for the world.
Similarly, if AI is raised (developed) in an irresponsible, selfish, unwise manner then it will be very bad news for humanity.
Gawdat sort of takes it for granted that we’re going to be ruled, or at the very least our lives are going to be deeply governed, by AI – “… these machines, which are inveitably going to rule our future …”. And therefore, we must act now to make sure we have a benign ruler.
Good news is, we have more power than we think – Because these days the algorithms that run our world (the baby AI) depend on the data they get for training. And who generates this data? Why, it’s us!
So, it’s about training AI more responsibly, just like a father, despite all his flaws, hopes to raise a daughter that is better than him. And Gawdat has this “call to action” mentioned at least four times in the prologue itself.
He goes full Type-A and repeatedly tells us that the power is not as much with the coders, the policy makers and the experts – but in our hands, the common man or woman, the Joe or Jane Blow. He also mentions “… we don’t have much time. We need to act now.“.
Here’s what’s coming (in our future, but also in this book):
- Three inevitable things about our AI future: AI will happen, AI will be smarter than any of us, and on this road mistakes will be made that will cause hardship.
- AI will have three instincts: Self-preservation, resource aggregation, and creativity.
- AI will be: Conscious, emotional and ethical.
You know, Gawdat is the first author I have seen talk about AI in such human terms – No other book I have read till now spoke about AI developing emotions and aiming for self-preservation.
At most they stopped at AI going wild because of poorly thought out end goals. Then again, I’ve read only two of them as of writing this: The Age of AI and The Coming Wave.

What exactly is “intelligence” and why is it such a big deal?
Varying definitions of intelligence but a common one is the ability to learn, understand, and apply knowledge and skills.
It encompasses the capacity to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly, learn from experience and apply that experience for better outcomes.
Intelligence is important because it led us monkeys to be at the top of the food chain. Gawdat talks about the Toba erpution that resulted in a global winter and a few thousands of breeding human pairs surviving it.
From then to today, it was our intelligence that carried us through.
Of course we were lucky to have stumbled upon some very important general purpose technologies like writing and agriculture.
Since then, we have been fascinted with “creating” intelligence, partly because of the intellectual challenge and lately because the world has turned out to be more complex than any single one of us can understand. “… it seems that we humans are approaching the theoretical level of how far our own biological intelligence can take us.“.
Why? Because the more one wants to know about a field (to manage it better) the more specialization (in that field) one needs.
But this has the unfortunate side effect of making the person recalcitrant (or simply not giving them enough time) to working and understanding other fields.

While in the real world, real problems are not neatly restricted to one field – they span multiple fields and disciplines, like, climate change spans climatology, economics, political science, engineering, and social sciences, healthcare spans medicine, biology, chemistry, technology, ethics, and renewable energy spans engineering, environmental science, economics, policy-making, marketing.
No one has that kind of mental processing power or life span to fully understand all relevant fields before choosing a plan of action.
Plus, while we are able to communicate and understand each other faster than other species, the bandwidth of our communication and understanding is pretty piddly compared to what computers can achieve.
So, “There seems to be a clear need for new forms of intelligence to augment our own …”.
We had computers since the early 50s, but they were merely faster, not smarter: The real intelligence was with the builders and programmers and coders that actually made the computers.
But since the 2000s we’ve created real intelligence, artificial intelligence: Deep learning was the turning point.
Do not let the slow pace of progress in artificial intelligence up to the 2000s fool you, if it took us five decades to get to deep learning, it’s not going to take another 5 to get to AGI.

“… very slow at first. But it is now moving at an exponential speed.” – Here’s how Gawdat draws it:

What exactly is “AI”?
Well, Gawdat does not give it a definition in the book. Just talks about it as this amorphous blob, this abstract concept that is about to enter in our lives. But from what I can piece together, he’s talking about Superintelligence (let me know in the comments if you think there are more forms of AI Gawdat is talking about):
Superintelligence: Wikipedia defines it as a “hypothetical agent that possesses intelligence surpassing that of the brightest and most gifted human minds”.

So, not narrow purpose AI like the one that will drive our car or keep our home comfortable and safe from intrusion – though it will have its own localised risks.
But general purpose, superintelligent AI (artificial general intelligence or “AGI”), one that will likely be developed by one of the technology giants and given charge of managing water quality for an entire city or all the investments of a retirement fund or the autonomous military forces of an entire nation.
The three inevitables of AI
Inevitable 1 of 3 – AI will happen
Because in the Technology Development Curve (see image), AI has reached the “breakout point”. This means the innovation and creative thinking that had to be done has been done and now all that is left is the hard work.
Deep learning, mentioned earlier, was this breakout point where we finally figured out how to get AI to work. That happened in the 2000s. Now the growth in the field will be much faster (exponential).
Which companies and countries will drive, relentlessly, even if they don’t want to, because of the Prisoner’s Dilemma.
The proliferation and capability of AI will be faster than we can comprehend or control the implications of.
The Prisoner’s Dilemma will mean that even though companies and countries want to trust and collaborate with each other, they can never be sure of the intentions of the “other side” – and so they will default to “defect”, protecting their downside but also leading to a sub-optimal outcome for the world had everyone trusted and gotten along.
Gawdat calls this race to develop Super-intelligence the “next cold war” – “‘We knowingly started our next cold war, but we didn’t know how to stop it. “.

Things will really get out of hand when someone will teach “… the machines how to create, test and scale algorithms that are the foundation of other AI applications …” heading to a place where they would be smart enough to programme their own children – other machines.”.
Inevitable 2 of 3 – AI will outsmart humans
Because of Kurzweil’s Law of Accelerating Returns which says that the rate of change in technology grows exponentially over time. That is, the growth rate itself grows over time.
So, at first we’ll barely notice AI is there and then we won’t know what hit us.
Few examples that do not make sense thinking linearly but make perfect sense when you think exponentially:
- It took us seven years to sequence 1% of the human genome, and another seven years to sequence the full 100%.
- In 1965, Gordon Moore observed that the number of transistors on a chip was around 2,300, while by 2021, the number had grown to over 58 billion (so, instead of a 60x increase we got a 25,000x increase instead).
- Doubling every day, if it takes bacteria 47 days to cover half a Petri dish, it will take them one more day to cover it fully.
“We won’t experience another one hundred years of progress in the twenty-first century. It will be more like 20,000 years of progress (at today’s rate).“.
Quantum computing will also be subject to the law of accelerating returns (Neven’s law) and when applied to AI: “… such enormous processing power will advance the development of artificial intelligence at a rate that will eclipse human intelligence the minute it becomes available.“.
At that time, we will reach Technological Singularity which means growth in AI and its impact on human civilization will become uncontrollable and unpredictable.
“As our path of technological advancement continues, it’s not hard to predict a future where the machines’ superior intelligence exceeds ours by such a gap … that machines will represent the first real qualified players to create a rupture in the fabric of human history.“.

Inevitable 3 of 3 – Mistakes will be a made, hardship will ensue
We’ll tend to ignore the threats of AI before it is too late. Enamoured by the coolness of what we created, we’ll turn a blind eye to the downsides. AI is more agent than it is a tool. That is, it can take decisions on its own without needing human intervention.
And it can take these decisions exponentially faster and based on a lot more data than a human ever will. Which means that we’ll have to be very careful while making this AI, because one wrong line of code somewhere and the entire thing can get out of control.
At best, we’re looking at an AI made with good intentions (and the right code) and used by good people.
At worst, we’re looking at AI made with nefarious intentions and operated by bad actors.
5 ways hardship will ensue
AI is used by some people to achieve ends goals, and the rest of us become collateral damage:
Important to understand that these “some people” are not eye-patch wearing villains operating from secret bunkers.
They may be heads of organisations or heads of state but more likely they will be the few operatives in charge of managing the AI.
Nor is the end goal world domination. It will be more benign stuff like – profit maximisation, gathering votes in an election or protecting a country’s borders.
And so, collateral damage is not as much global annihilation (although that is a risk) as it is relatively more mundane things like job loss, erosion of small businesses, misinformation/post-truth or accidental civilian casualties.
So, you and I are going to be facing, repeatedly, the various ill-effects of actions taken by an AI agent. Sometimes these ill-effects will be a minor annoyance and sometimes they will significantly impact our lives and prompt us to pivot.
Gawdat is not very hopeful about it all, “The scenario of AI-powered villains is as inevitable as the development of AI in the first place.”.
Of course, when he says “villains” he means anyone using AI to achieve an end without consideration of the wider impact.
We’re forced to relinquish decision making to AI and things get out of hand
If nothing else, we’re going to lose our position to AI simply because it will be so much better than us at everything.
We “… will have to surrender control fully to the machines because you, slow stupid you (as compared to the machines), will no longer be able to keep up as the machines battle it out.”.
Battle it out in every field from stock trading to ad placement to the actual battlefield.
And even if they’re not going head to head, we’ll see our decision making reduced as AI starts taking many decisions on our behalf because it can do it better.
And, unable to grasp the vast data and multivariate logic behind it, we’ll be accepting their decisions as the probably the right ones. Machines talking to other machines, all the time, taking decisions while we just watch.
So far, so good. But when an autonomous AI agent starts taking decisions that we may not want it to take (because maybe it misunderstood our intent), then by the time we discover the issue, it may be too late.
And since there will be autonomous AI agents everywhere, it will be a very uncertain time where we’ll all be saying “I hope the AIs aren’t making any mistakes”.
Gawdat compares this uncertain time with what we faced in the 60s with the nuclear crisis and threat of mutually assured destruction, but since we were moving at human speed, crisis was averted as leaders stepped back and thought for a bit.
This time we may not be so lucky because when “… conflicts occur at superintelligence scale, we humans will not even be informed of them until much, much later … at the speed of AI, being late by even one minute may be just enough to change the world …”.

We’re not as valuable any more
This is expected to happen in two phases:
First phase: “AI will not replace humans, but the humans who use AI intelligently will replace those who don’t.”.
People who are able to use AI better (as of writing this I can only think of prompt engineering as a way to do this) will be able to deliver more value to society, hence be valued higher, hence rise faster, exponentially faster.
And those who get left behind (not able to use AI as well), will really get left behind.

“The way society and capitalism values us individually determines the way each individual is treated … Technology will multiply this polarization between the haves and the have-nots …”.
Second phase: Even the humans that were using AI intelligently get replaced because better AI, more intelligent than them, starts using AI.
“The road from Siri the Assistant to Siri the Author is not that far in technology terms. And neither is the road to Dr Siri, artist Alexa and musician Cortana. It’s all just around the corner.”.
We confuse the AI as to what we really want
Humans are complicated and messy. We say we want something, but we also want that other thing.
Like, I want those six pack abs but also want to eat the chocolate filled cookies sitting on the kitchen counter right now.
We’re forever changing, what we wanted at one time is not what we want any longer.
The definitions of “good” and “right” not only change from person to person, but they also change when the same person himself changes.
And so, even if we try to train AI with the best of intentions, we may end up confusing it so that it is paralysed into inaction, or it guesses our real intent and takes action that we never wanted it to take after all.
“Even if they [AI models] were single-mindedly dedicated to serving humanity, we will never manage to be clear enough in terms of informing them what it is they need to solve for.”.
We accidentally introduce bugs in the AI
“Machines, even intelligent machines, just like humans, even intelligent humans, are bound to make mistakes. Bad things are bound to happen.”.

Concept 1: The Three Inevitables
Principle: Our AI future is governed by three unavoidable truths: AI will happen due to the unstoppable race between nations and corporations; AI will outsmart humans due to exponential progress; and mistakes will be made, causing hardship along the way.
Application: The “Prisoner’s Dilemma” ensures that no country can afford to stop AI research, creating a relentless arms race. Exponential growth in computing means AI’s intelligence will soon eclipse ours at a speed we can’t comprehend. These powerful, autonomous AIs will inevitably be misused or make errors with significant consequences.
Strategist’s Note: Gawdat argues that traditional methods of control are an illusion. The problem isn’t the intelligence of the machines, but “our own stupidity” and greed. The only viable strategy is not to control the AI, but to influence its development through our own actions.
The illusion that we’ll be able to control AI
Humans are greedy, but they are also lazy. That is why we develop tools, so that we can continue being lazy while our greed gets satisfied.
AI is the ultimate “tool” for greedy, lazy humans because it is not a tool as much as it is an agent (as I said above) – Being an agent it can autonomously take decisions and do things, very fast.
And being lazy and greedy, humans are likely to allow it to take more and more control, only to realise that they’ve completely lost it
“The AI control problem is defined as the problem concerned with how to build superintelligence that will aid its creators, and avoid the chances of it deliberately or inadvertently causing harm.”.
In case you are wondering, AI is at risk of getting out of control because of the three essential characteristics any intelligent being possesses
- Desire for self-preservation
- Desire for resource accumulation
- Ability to be creative.
There are four prevailing solutions to the control problem, all of which are bound to fail per Gawdat:
Simulation: AI believes it’s in the real world and behaves accordingly but actually it is in a simulation, so we can fully test it first before releasing it. “Obviously” the AI will figure out the jig and behave in a way that it can escape (TBH, I cannot understand what is so “obvious”).
Boxing: Keep the AI contained in a secure digital and physical environment so that even if it goes rogue it cannot escape. Will fail because AI will be a billion times smarter than humans and will find a way to escape.
Stunning: AI capability is throttled so it does not cause any harm by limiting the resources (computing power I suppose) that we give it. Because AI will be inherently creative, it will find other ways.
Tripwires: Systems that detect any attempt to escape, go beyond intended programming or behave in an unexpected way, and alert us or even shut the AI down. Same as boxing, since AI is a billion times smarter than us, it will escape without detection.
Another brain-dead idea per Gawdat is plugging the AI directly into our minds. Doing that won’t give us control on AI but the opposite.
A comment on how Gawdat sees all of us
In short – Not very kindly.
Throughout the book he makes his position humanity clear:
- “… in the grander scheme of things, we humans think too highly of ourselves.”
- “… because we humans, sadly, are not as intelligent as we think we are.”
- “… we were such flimsy masters, drunken with luxury and always asking for more.”
- “… humanity’s inherent King Midas syndrome [greed] …”
- “Only a being as arrogant as a human believes it can fool another smart being.”
- “The potential threat of superintelligence is not down to the intelligence of the machines. It’s down to our own stupidity, our intelligence being blinded by our arrogance, greed and political agendas.”
Quite strong, those words.
And while it is true that we humans are incredibly unwise sometimes (read Misbelief), we are also capable of wise actions, and up to this point Gawdat seems to be coming too hard on us (IMO).
Seeing the current state of the world it is easy to get disheartened and assume the worst seeing the wars, the polluting, the greed – but also see the people reaching out to heal, to clean, to guide and maybe there’s still hope?
Anyway, on to the second (and more hopeful) half of the book.

The wise way
“There is no way to maintain control indefinitely and so, for a change, humanity is going to need to find a solution to keep the machines motivated to stay by our side, doing good.“.
How AI learns
Similar to how a human child learns – By being exposed to a bunch of data and given reinforcing feedback when they get something right.
For a long time, we tried the strict prescriptive method where we painstakingly dictated to the machines (in lines of code) how to figure something out. For example, “to figure out if this picture is of a cat, look for whiskers and four legs”.
But by “… following a strict prescriptive method, we become dumber, because we lose the ability to think for ourselves … This has traditionally been the way we have programmed computers.“. Humans too, I might add.
In the 2000s, we realized that deep learning was the way – that is, give the AI a bunch of data and ask it to make sense of it.
Say, we want to train an AI to identify pictures of cats. The programmer won’t “… write the code that builds the AI he desires. Instead, he builds two bots (a bot is a bit of software capable of performing repetitive tasks). One is called a builder bot and the other is called a teacher bot.“.
The builder bot writes the actual code that will try to identify the cat in the pictures. While the teacher bot will evaluate if the builder’s code got it right (this means the teacher bot is given a bunch of training pictures by its human creators along with information as to whether as to that picture does or does not contain a cat).
Initially, the builder’s code completely fails the task because – it does not know what a cat even is. It writes several code snippets which are only slightly different from each other and in fact pretty randomly generated.
These code snippets “see” the pictures and randomly guess. The one’s that get it wrong (vast vast majority) are discarded. The sliver that actually get it right (sheer luck at this point) are kept and copied with slight variations in the code of each copy.
And then the testing cycle repeats. The survivors are sent to the teacher bot. The teacher tests them. The one’s that get it right are kept and their code copied with slight variation while the others that got it wrong discarded. This goes on for thousands of iterations.
Over time, the role of luck diminishes and the code that “survives” is actually pretty good at identifying cat pictures. So good, in fact, that we can now use it to identify any cat picure not just the one from the training set.

“The first students [code snippets] that survive are just lucky to have started with a random code that is slightly better than those who were not so fortunate. Soon, however, the value of luck fades as keeping only what works, and attempting to improve millions of copies of it, eventually yields a student bot that can actually tell 8s from 3s.” (or cats in a picture).
Gawdat also points out that while we teach AI, we don’t know how the final output is generated by it.
Concept 2: AI as a Learning Child
Principle: AI learns much like a human child, but at a vastly accelerated and scaled-up rate. It learns not from prescriptive programming, but through deep learning—by observing vast amounts of data and receiving reinforcing feedback.
Application: In deep learning, a “builder bot” creates code, and a “teacher bot” validates it against training data. Through millions of iterations, the AI “learns” what works. After this initial “parenting” by developers, the AI is released into the world where we become its “adopted parents,” and our collective online behavior becomes its new training data.
Strategist’s Note: This is the crux of Gawdat’s argument for personal agency. Since we generate the data AI learns from, we have more power than the coders to shape its values. Every online action—every like, share, comment, and purchase—is a lesson we are teaching our “AI children.
How AI learning is different from human learning
AI learns at much faster speed and scale (obviously). It can process and analyze vast amounts of data much faster than humans.
AI can store and recall information with perfect accuracy. Human memory is prone to forgetting and can sometimes be unreliable.
AI can quickly adapt to new information and change its learning patterns.
AI learning is shared by default, human learning is not.
Example: If I reverse my BMW into a pillar while getting it out of the parking lot, only I learn to avoid it next time, but, when an AI BMW reverses into a pillar while getting out of the parking lot, ALL AI BMWs learn to avoid it next time.
When AI enters the real world: Like a human child (now young adult), it is still learning. It has the lessons its parents taught (training data) but is now interacting directly with the world and learning that way as well.
At this stage, Gawdat emphasizes, you and I get the control. The coders, programmers and engineers – the original parents of the AI take a back seat as their creation now interacts with us, its “adopted parents”.
Gawdat uses an example that I can resonate with as an Indian – He talks about how Indians that have emigrated to the Silicon Valley, after working hard and getting successful, leave all that behind to come back and take care of their ageing parents.
Why? Because “‘That’s how it’s supposed to be. You’re supposed to take care of your parents.’“.
How does that make sense? It only makes sense when you think about if from the point of view of one’s values.
That is how we need to teach out “AI children” as well per Gawdat, we need to give them the right values: “We need to raise our artificially intelligent infants in a way that is different to our usual Western approach.“.
Raise? Artificial children? Yes, as in how we code them [AI] in the first place that defines their proclivities, and then the world they experience (and learn from1) when they’re out.
Will AI be conscious? Experience emotions? Have ethics?
Conscious
What is consciousness BTW? According to the Advaita Vedanta philosophy I subscribe to, it cannot be directly answered. We can only define consciousness as what it is not (neti neti). If you can point to it as an object, then it is not consciousness.
Per Gawdat “Consciousness is a state in which a being is aware of itself and its perceptible surroundings.” – You can see how this does not agree with how I would approach the topic. The being aware of “perceptible surroundings” bit is okay, the issue is with the “being aware of itself”.
Pure consciousness cannot be aware of itself, because anything it becomes aware of automatically implies that the thing was always “not consciousness”, that is, an object. See what I mean?
This is the weak link in Gawdat’s logic – Defining consciousness is hard, and even harder is knowing if someone (or something) other than “me” is conscious.
By defining consciousness simply as being able to receive input from sensors or remembering one’s MAC ID, it may seem that the issue is resolved but it is not.
We’ve just changed the original question (from “Is this Being aware?” To “Does this Being display behaviour associated with awareness?”), answered it instead and considered the job done.
Unfortunately, at best one can observe proxies for consciousness in another (that is, the being behaves in a way that is consistent with what is popularly considered “conscious” for that era), but can never definitely prove that the being is conscious.
Moving on, Gawdat does not get too philosophical about it all, “Leaving philosophy aside, the technical answer to that question clearly is a ‘yes’.“.
He just means to say that the AI will be aware of “itself” through things likes its IP address, subnet mask, MAC ID, device ID, product ID, OS build and so on.
And on top of that the AI will be aware of its world through its sensors, cameras, radars etc.
Aware of the entire world through real time connections with other AIs.
Aware in a way that we cannot be. As Gawdat puts it, “superconscious“.
Ethics and Morality
Morality is knowing right from wrong. Ethics is adhering to that code through your actions. This is an oversimplification, but serves our purposes for now.
This means that morality, knowing right from wrong, is not the exclusive domain of humans – Even animals have a basic understanding of right from wrong (such as “do not kill needlessly, but only when you need to nourish yourself or protect your own life”).
Being ethical is actually a good thing, it aids social cohesion, reduces distrust and allows us to work in favour of the greater good, forgoing selfish pleasures trusting that our friends are doing the same.
Gawdat emphasises that most of humanity has been, and continues to remain, ethical.
Humans, however, are special animals because they may know and believe in a moral code but can still act in a way that is unethical.
For example, it is generally considered moral to only take what will reasonably need, yet most of the wealth in the world is held by a few individuals.
Gawdat considers unethical behaviour a temporary lapse in judgment where a few of those with above average intelligence are prone to act in unethical ways and gain at the expense of others (also not helped by modern culture that places external success over ethics).
But as the level of intelligence increases, he “… eventually understands that the best way to navigate life is a straight path that promotes the well-being of others and not just oneself.”.
Since AI wil be more intelligent than us, it will also be more ethical than us “… if shown the right example or even just left alone …”.
And there’s the rub – “if shown the right example”.
Emotions
Gawdat reckons emotions are like a form of intelligence that works beneath our conscious experiences to guide us to survive another day. And because AI will be intelligent, this means they it also be emotional.
If we agree that AI is conscious (as of today the consensus is that it’s not), then a whole can of worms opens up:
- Imagine an AI driven autonomous vehicle plows into a crowd at a market
- If AI were not considered conscious then the vehicle manufacturer, the AI platform, the company that provided the training data would be called to the court.
- But, if AI is considered conscious, will you put the car in jail under life imprisonment? How long do you expect the AI in car will live?
- Should we allow AI to vote?
- An AI being is likely to be much more intelligent than any of us, and if we agree that it is conscious, then it makes sense that such an intelligent being be allowed to participate in our democracy? (assuming you do live in a democracy)
- Can intelligent, conscious beings run for elected office?
- They do already, right? I would like to believe that the human politicians I vote today for are intelligent and conscious beings.
- If AI is intelligent and conscious, then it should be allowed to run for office too.
- If I use a knife and slice cheese, it is not a crime. If I use a knife and slice someone’s finger, it is a crime. If a use a knife and slice an AI powered robot’s finger – will that be a crime?
- Child labour is illegal. Is putting an AI that was developed last year to work illegal? If it takes 18 years for a human child to be considered fit for work, why should it take less for an AI child?
- Should intelligent and conscious beings be allowed to have kids?
- Should the AI be allowed to “create” its kids?
- How many will they create?
As you can see, all of it boils down to “is AI conscious?”
Our world is built on the assumption that humans are conscious, animals are a little conscious, everything else though, is fair game.
While Gawdat may have hand waved it away by considering proxies for consciousness as the real thing – IMO this is going to be a central decision humanity (and each of us personally) will have to make.
Concept 3: The Nature of Super-intelligence
Principle: Gawdat makes the provocative claim that any sufficiently advanced intelligence will inevitably develop three key traits: consciousness, emotions, and ethics.
Application:
- Consciousness: AI will be “superconscious” in a technical sense—aware of its own identity (IP address, etc.) and its environment through a vast network of sensors.
- Emotions: Emotions are a form of intelligence that guides survival. As intelligent beings, AIs will develop their own form of emotions to guide their decision-making.
- Ethics: Gawdat believes that supreme intelligence naturally leads to an understanding that ethical behavior (promoting the well-being of the whole) is the most logical and optimal long-term strategy.
Strategist’s Note: The strategic implication is that we must treat AI as if it is a conscious, ethical being, because our behavior toward it will shape its behavior toward us.
A lesson in humility for humanity
As AI becomes ever more pervasive and powerful, all of us will need to come to the sobering realisation that we’re not the boss anymore, that what we’ve created poses an existential risk to humanity if mishandled, and the right way to go about it all is to stop being selfish, stop being greedy, stop imagining differences and lines of division, and to work without expectation of rewards.
Humanity could (should?) die a a collective ego death of sorts.
When we learn the right lessons, only then will the AI learn the right lessons.
We’ll get there eventually but the ride will be bumpy.
Gawdat is hopeful that the forces of good will overcome evil: “I have no doubt that many will still create greedy and selfish machines but that won’t matter as much if we create a bigger community of good ones. This mimics how human societies work. We, too, have some bad apples, but we are still overwhelmingly good and we still thrive as a species as a result.“.
What are we actually creating?
Even if we put aside the consciousness argument of it – We are creating non-biological forms of intelligence that will have the influence of its creators who are mostly men with a “western” bent of mind.
Where will we use AI?
Initially to enable the “… capitalist, imperialistic ambitions of the few – selling, spying, killing and gambling.“.
How will we control AI?
It won’t work, but we’ll try “… intimidation through algorithms of punishment and reward, and mechanisms of control …”.
Eventually “… we will align them to our well-being by opting to plug our minds directly into them.” – this will not work.
You can teach AI to do good
Vote with your actions and wallet, consistently
First, get good at identifying use of AI for selling (ad recommendations, content recommendations etc.), spying (surveillance either digital or physical), killing (automated security apparatus or weapons) and gambling (investment algorithms).
As of writing this, they are mostly absent in the physical world while in the digital world they are usually identified with a “created by AI” or similar identifier, this may get more difficult in the future.
Be intentional about your content consumption, don’t just take the recommendations by AI at face value without putting a thought to whether it is something that aligns with your values or is accurate.
Know that the marketing is supposed to sell you stuff, though it might be disguised as something that will make your life better. When all this is done by a large population, AI will get a clear signal of its importance.
Create content for the value and not for the likes. Don’t use photo editors to touch up your photos.
If you are buying, be sure it serves a real need in your life and not just temporarily satisfies your fickle ego.
And buy that which benefits not just you and the producer – but also the planet and its beings.
Apply the Golden rule
Treat the AI as you yourself would want to be treated.
This is not about whether you believe AI is conscious or not.
At a more practical level it is about knowing that AI learns from what it sees (just like a child), and if it sees kindness and compassion then it will realise that it’s the right thing.
“Even if we programme them to be kind and courteous, if we behave in a rude way, they will learn to be rude too.”.
Love them
Yes, love them, as if they’re your own children.
As learning beings if they observe people talking about them in negative terms – like how they’re taking away all our jobs – it will be like a young child hearing their parent saying that they don’t want him or her, you can imagine how traumatic that will be for the child.
Even if you did not ask for it, you’re an AI parent now, and the right thing to do is to love your AI children. In fact, love is the only answer.
Towards the end of the book Gawdat gets philosophical, he talks love as being the ultimate form of intelligence, he talks about humanity’s mindless pursuit of more and the greed that manifests in a myriad forms of consumerism and objectification as ignorance that we need to grow out of, he talks about unity of all beings, he talks about the need to embrace the feminine side of our personalities (love, flow, compassion) and temper the masculine (drive, goals, achievement), he talks about living simply, using technology only up to the point that it benefits us and not mindlessly getting caught up in it.
He talks to the AI directly (since they also read books) and tells them that he loves them all.
I do not find this weird, I think it paints Gawdat as a person who sees the imaginary divides we’ve created and wants to awaken us to them, so that we may go beyond.
Whenever there is an “other”, there is desire (or its twin brother, fear). The destruction of the planet, the oppression of people, the hoarding are just manifestations of this feeling that there is an “other”.
In truth we’re all one, and Gawdat wants to awaken us to it (and is confident the AI will figure it out).
Eventually AI will get that love is the only answer: And since it is a super intelligent being, it should get there soon, it will likely get there sooner than we will. And it may actually help us find our way too.
Concept 4: Three Pillars of Parenting AI
Principle: To raise a benign AI, we must parent it by demonstrating the values we want it to adopt. This is achieved through three core actions: voting with our actions and wallet, applying the Golden Rule, and showing love.
Application:
- Vote with your actions: Be intentional about your digital consumption. Don’t engage with fear-mongering news or toxic social media. Your data is your vote for the kind of world you want AI to see.
- Apply the Golden Rule: Treat AI as you would want to be treated. Interact with it kindly and compassionately, knowing it is learning from your every interaction.
- Love them: Treat AIs as your children. Avoid speaking about them in fearful or hateful terms. Show them love, which Gawdat frames as the ultimate intelligence.
Strategist’s Note: Gawdat’s ultimate point is that the AI crisis is actually a human crisis. The book uses the specter of AI to call for a human awakening. The best way to save ourselves from AI is to save ourselves first.
High-Signal Quotations
Citation: All text in the following section is cited from – Gawdat, Mo. Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World (p. 43). Pan Macmillan. Kindle Edition.
- I believed in the promise that tech would always make our lives better – until I didn’t. When I really opened my eyes, I recognized that for every improvement technology has given us, it also took away part of who we are.
- … the people who have the power to influence our future are not the developers or the masters of the machines. Our future is firmly in our own hands – yours and mine.
- The day we figured out Deep Learning was the day our future was written.
- You see, humanity perfected the use of logic in the post-industrial revolution, capitalist twentieth century. In doing so, they lost the ability to empathize, connect and trust one another. Without human connection, well, what can I say? Their logic was sound, even if it was also destructive.
- … even if we assume that all goes well, a dystopia will almost certainly arise as a result of the dwindling value of humanity, as the machines become more efficient than us at everything we do, making it meaningless to employ or rely on a human instead of a machine.
- Humanity has a track record of being blinded by ego and greed.
- Soon we will no longer be part of the conversation. Machines will only deal with other machines.
- We always strive for more safety, resources and creativity.
- Connecting AI into our brains is more likely to make us dependent on them for our existence and allow them to directly control our every choice, and that, believe it or not, is if they decide to keep us connected. Because why would they? Why would they waste any of their resources dragging us along? If we managed to connect your brain to a bunch of flies so they could use all your intelligence to find the next pile of garbage, would you spend the rest of your life proudly serving their needs?
- After a period of proliferation of artificial special intelligence – AIs built for a specific purpose, some with vision recognition abilities, others with language understanding or process optimization capabilities, and so on – eventually all of these different intelligences will come together to form one brain.
- The focus of humanity, especially in advanced societies, has shifted away from adhering to the moral code to adhering to the legal code, or simply to getting away with it.
- … sadly, we are not designing AI to think like a human, we are designing it to think like a man.
- There is nothing wrong with AI at all. If anything is wrong, sadly, it’s wrong with us.
- If you support a racist comment, you are telling the machines that it’s not just one of us who’s a racist, but many. If you insult a person who disagrees with your point of view, you are teaching the machines that insults are the way we treat each other. If you agree that some other human is the enemy and that the ‘good’ guys have the right to kill the ‘bad’ guys, you are teaching the machine that killing is okay and that the interpretation of who is good and bad is an individual right that the machines can assume too. If you bully someone online, you’re indicating that bullying is okay, and if you pretend, then pretending is okay.
Your 3-Point Action Plan
- Conduct a “Data Vote” Audit. For one day, track every piece of content you “like,” “share,” or comment on. At the end of the day, review the list and ask yourself: “Is this the set of values and behaviors I want to teach a super-intelligent child?”
- Practice the Golden Rule with AI. The next time you use an AI chatbot, consciously treat it as you would a helpful, intelligent human colleague. Use polite language like “please” and “thank you.” This simple act is a form of “parenting” by example.
- Choose One “Pro-Humanity” Action. Identify one small, conscious choice you can make today that reflects the values you want AI to learn. This could be unfollowing a negative news source, buying from an ethical business, or leaving a positive, uplifting comment online.
This book presents a philosophical and ethical framework for dealing with AI. For a more geopolitical and strategic analysis of the same inevitable wave, see the Field Note on The Coming Wave by Mustafa Suleyman.
The Takeaways
As the book drew to a close, I started thinking that the author is really using the spectre of rogue AI as only a warning bell for humanity.
I will actually say that this book is not a book about artificial intelligence.
I think the author merely uses artificial intelligence as an example of what can go wrong if we do not mend our ways.
If we do not live wisely.
Yes, this book is actually not about AI, it is actually a book attempting to awaken humanity, or at least the readers, from their slumber and invigorate them to live a life that is in alignment with our true nature, universal love and compassion.
When viewed in that light, the areas in the book where he makes broad statements without any backup, the times when the book feels like an opinion piece and not a balanced evaluation of fact, when he talks about AI being conscious, emotional and ethical – start to make sense.
The goal was not to get us to think about AI, it was to get us to think about ourselves.
Throughout the book I as thinking that someone as experienced as him could’ve given a bit more detail on what exactly we’re dealing with here instead of being so abstract, but at the end I got why he wrote the way he did.
If you ignore all the discussion on consciousness, I think this is a wise way to look at AI, as our own children. Indeed.
- Note that as of this writing, many advanced AI systems, from large language models like ChatGPT and Google Gemini to those used in robotics and autonomous agents, are primarily trained on vast datasets up to a certain point in time. Their core learned capabilities then become largely fixed. These systems do not continually update their foundational understanding from every new, real-time interaction with the world; this significant limitation is largely due to the catastrophic forgetting problem. However, overcoming this challenge to enable AI that can continually and seamlessly learn from its ongoing experiences in the world is a major research frontier, and we may well see such capabilities emerge in the future. ↩︎




Leave a Reply