AI Snake Oil by Arvind Narayanan and Sayash Kapoor

“AI Snake Oil is AI that does not and cannot work as advertised.“.

In this definition the authors call out both the possibilities: AI that does not work, as of today, work as advertised, and AI that may never work as advertised due to inherent systemic limitations.

In a subtler way they also point out to us that AI is not a single thing. It is a common term used for a collection of technologies, fine for a conversation around the water cooler but if you’re discussing AI in the particular contexts then it will be better to be specific about the AI you are talking about.

Examples of AI include Generative AI, Predictive AI, Content Moderation AI – which also happen to be the AIs examined in the book – but the authors also make it clear that a precise list is not present of what all constitutes “AI”.

However, in everyday situations we are likely to encounter the aforementioned AIs, so the list while theoretically incomplete is practically adequate.

Is AI snake oil an honest mistake or a deliberate ploy? The authors opine “Commercial AI snake oil usually results from companies knowingly selling AI that doesn’t work, rather than they themselves being fooled.“.

In the book the authors also present the landscape of AI snake oil, plotting various technologies like code generation and facial recognition on a scale going from “Works” on one end to “Snake oil” on the other. Consider buying the book to learn more and to support the authors.

It also makes sense to define what “AI” itself is, since the term has an aura of mystery these days and in some places may actually serve as a “Get out of jail free” card:

• Manager: How exactly did you come to the conclusion that the demand for the product is expected to rise precisely by 5.443564% next year?
• Subordinate: Oh, the AI model gave me that answer.
• Manager: Sounds good!

Recently the prices of “.ai” as a TLD (top level domain, such as http://www.xyz.ai) have spiked, I imagine that at least some of this demand is linked to the hype around AI.

When “the AI said so” becomes a substitute for reasoned explanation, critical scrutiny, or transparent methodology, it bypasses accountability and can lead to all sorts of unexamined decisions – the environment where “snake oil” can thrive.

So, according to the authors, one can call the technology “AI’ if it passes the following three tests:

1. Does the task need training and creativity for a human to do? If yes, and a computer can do it then it may be AI.
2. Was the behaviour directly programmed or did it emerge? If it emerged then it may be AI.
3. Can it make decisions/adapt to its environment autonomously? If yes, it may be AI.

1. Approach 1: Symbolic machines i.e. teach the machine how to predict
2. Approach 2: Machine learning i.e. let the machine learn how to predict

Approach 1: Symbolic machines

Here you give the machine immutable laws and equations that describe the relationship between the input and output, and so it will be able to predict (but really, calculate) the future.

Predicting physical phenomenon generally fall under this category, called “simulations“, such as predicting the weather, planetary movements or seismic activity. Even when applied to the social sciences or economics, simulations are useful at a gross level.

This approach can work when either:

• You know precisely the relationships between things and thus are able to code that in (like a completely closed system of your own design), or
• If you’re looking for output at a high level where accuracy is not that important (and so you focus on only Pareto inputs), or
• Where you’ve done some sort of “phenomena level matching” and lower level inputs are subsumed in higher level inputs thus allowing you to predict a similarly higher level output.

Approach 2: Machine learning

The other approach is to give all the data to the machine, at the highest resolution of detail, allow it to figure out the hidden patterns and then become a prediction machine, i.e. “machine learning”.

ML thrives on having the right quality and more-the-better quantity of data.

As a system gets bigger, its complexity increases exponentially, which means you need exponentially more data to predict things.

That is why, unless you have a lot of training data, ML is better for predicting things at a simpler level.

Simulation software

Simulation software is based on the principles of symbolic machines, which means that the cause-effect relationships are coded into the software by humans. This needs experts, and the scientific community today is mature enough that if you call yourself an expert your thesis needs to be proven and vetted by the scientific community.

Even before you touch a computer, you first have to prove that your theory holds water in being able to make accurate predictions, you have to be able to explain to a reasonable degree how your theory works and why it works.

And thus by extension the coding that you do in the software will be a somewhat accurate representation of reality and thus the predictions will be somewhat accurate.

This doesn’t mean its not possible to make poor simulation software, just that the chances of coming across poor sim software are less because the field is more mature, and the SW entirely made by humans so there less mystique around it and a a stronger pressure to explain and demonstrate predictive power.

Predictive AI

Here you are using ML to predict a dependent variable from an independent variable(s), by letting the AI identify the connections and weights from your labelled data.

Now, if your conceptual understanding of the subject matter is strong, then your training data will be correctly collected, cleaned, labelled and structured (i.e. your independent variables will have real causative effect on the dependent variable), and your ML model will also be correctly setup.

But if your understanding of the cause-effect relationship is weak, you may not even know whether the independent variable(s) you are offering as training data are the right ones or at the correct resolution. In this case there is the risk that your ML model identifies “imaginary” or “forced” connections, and seems to predict things but may just be doing GIGO (garbage-in, garbage-out).

And the more dynamic, open, complex the system you are trying to predict, the more nascent its field, the more tacit the knowledge of its experts, and higher the resolution requirement of your predictions (i.e. predicting at a local vs. a global level) the more likely you are to go wrong with Predictive AI.

Generative AI

Here too you are asking AI to predict using ML, but it differs from Predictive AI in two important ways:

1. In Gen AI, the dependent and independent variables are of the same “class” (e.g., using text to predict text), which may not always be the case with Predictive AI
2. Gen AI trains on unlabelled data (that it self-labels)

These two factors combine to eliminate the risk of imaginary or forced connections – and so, if a connection is identified in a Gen AI model then it is a real one (in the sense that it does exist in the training data and has genuine predictive impact).

This is the fundamental difference between Generative AI and Predictive AI (in how they have been defined in the book). Even though both work on ML principles – predicting in a generative sense is likely to be more robust, especially with larger pre-training datasets.

Plus, when it comes to the output of Generative AI, you are interested in only the possibilities and not a final, correct answer. This eases the expectations from Gen AI models.

As should be clear from the previous section, the likelihood of ML based AI being snake oil increases when you try to use ML to predict behaviour of dynamic and open systems, while not having sufficient training data and/or the conceptual clarity whether the training data has any relationship with the predictive goal.

The bottom line is, if your use of ML based AI is coming from a need to overcome a physical limitation (such as computational complexity) while your conceptual understanding is on point, then it may still work. But, if you are using ML as a stand-in for lack of conceptual clarity, then you will run into trouble.

If the underlying principles and cause-effect relationships of a system are well understood (high conceptual clarity), but modelling them with traditional symbolic methods is too complex or computationally expensive, then ML can be a powerful tool to approximate solutions or find patterns efficiently.

In many scientific and engineering fields, the fundamental laws might be known (e.g., the physics of fluid dynamics, complex biological interactions), but solving the equations for large-scale, real-world scenarios is intractable. In such cases, ML can definitely help.

Here, the ML is augmenting human expertise and well-grounded theory, not replacing its need.

The “guardrails” of conceptual understanding help in designing the ML system, selecting relevant features, and interpreting its outputs critically.

But, if ML is applied to a system where there’s little understanding of the underlying mechanisms, and the hope is that the AI will magically find meaningful relationships or “figure it out” without human guidance, the results are likely to be problematic.

ML will not magically tell you which data inputs (features) are actually relevant or if crucial ones are missing.

• Spurious Correlations: The AI might find statistically significant patterns in the data that are merely coincidental or due to hidden confounding variables, rather than true underlying relationships. Acting on these can lead to poor or even harmful decisions.
• Interpretability and Trust: If the model is a “black box” and there’s no underlying theory to explain why it’s making certain predictions, it’s hard to trust its outputs, especially in high-stakes situations or when the data landscape shifts.
• Generalisability: Models trained without strong conceptual grounding may perform well on the training data but fail to generalize to new, unseen situations because they’ve learned superficial correlations, not fundamental principles.
• GIGO (Garbage In, Garbage Out): Without conceptual clarity, it’s harder to even assess if the input data is appropriate or if the AI’s outputs make sense beyond fitting the historical data.

This has significant practical implications for decision-makers in both companies and governments. How an organisation approaches the use of AI – either as a targeted tool to overcome known limitations in a conceptually understood framework, or as a speculative substitute – will massively impact its success and risks.

For everyone

• Humility in the Face of Complexity: Decision-makers must recognize that AI is a tool, not an oracle. If the underlying system isn’t understood, AI won’t magically provide perfect clarity; it might just reflect our existing confusion with greater speed and scale.
• Value of Domain Expertise: The “death of expertise” is unlikely; in fact, deep domain knowledge becomes even more critical to guide AI development, validate its outputs, and understand its limitations.
• Iterative Approach: In domains where conceptual clarity is evolving, use AI as an exploratory tool to help build that understanding, rather than as a definitive decision-making engine from day one. Start with low-risk applications and iterate.

For decision makers in companies and governments

1. Strategic AI Investment:
   • Prioritise conceptually grounded projects: Focus AI investments on problems where there’s a existing understanding of the domain.
   • Augment expertise, don’t just replace it: Use AI to enhance the capabilities of skilled professionals by providing them with better tools for analysis or prediction within their understood domain, rather than building AI systems operating in a conceptual vacuum.
   • Be wary of “AI solutionism”: Scrutinise proposals that suggest AI can magically solve complex problems where your organization lacks fundamental understanding or data. If the problem isn’t well-defined or understood conceptually, AI is more likely to produce GIGO or spurious correlations.
2. Risk Management & Governance:
   • Assess conceptual clarity as a risk factor: When evaluating AI projects, the level of existing conceptual understanding of the problem domain should be a key risk assessment criterion. Lower clarity means higher risk.
   • Demand interpretability (where possible): For high-stakes decisions, push for AI models that offer transparency (XAI) or can be explained in terms of the business logic, especially if the underlying domain understanding is still developing.
   • Implement robust human oversight (Human-in-the-Loop, HITL): Especially in areas with lower conceptual clarity.

Having understood the concepts above, it should come as no surprise to you that Predictive AI is the field with the most amount of snake oil. “In contrast to generative AI, predictive AI does not work at all.“

To be fair, Predictive AI powers a lot of things – from your content recommendations (YouTube, Netflix, Spotify and so on) to stock trading algorithms to weather forecasts to healthcare software and more.

Predictive AI are models used for decision making based on predictions about the future.

“Models” are special cases of computer algorithms where the computer itself makes the algorithm for it to follow using machine learning.

And in many of those cases it does work (my YouTube feed does recommend what I tend to like).

Why then do the authors make such a scathing indictment of Predictive AI?

In my read of the book I felt that the authors are talking about a unique case of Predictive AI: when it is applied to predicting human behaviour.

Practically speaking, predicting the behaviour of incredibly complex (but more importantly, open) systems such as human minds may be impossible because we may never have enough data to feed into ML algorithm for it to be able to establish meaningful connections, sometimes called the “eight billion problem“.

But even that would be fine and predictions in limited contexts may still be possible (as is the case with music recommendation algorithms which also predict human behaviour albeit in a very narrow sense), if it weren’t for what I think is the real mistake AI companies make: training Predictive AI without having a strong conceptual understanding first.

The examples of Predictive AI snake oil cited in the book mainly are cases where AI companies train the tool on the customer’s existing data, without knowing the real driving equation/factors behind the predictive goal, or whether the customer’s data is even a correct proxy for the actual independent variable(s) that influence the dependent one.

Opacity of Claims

Naturally, if you don’t have conceptual clarity, your claims will be opaque and this is the first and foremost issue with Predictive AI.

The model design by AI companies is hidden as it is a trade secret, this is understandable. But when it comes to the claims these companies make about the performance of their AI software – even that is obfuscated behind asterisks and fine print.

When a Predictive AI tool makes a statement like “our tool was able to predict the outcome in 90% of cases” it is very important to know whether those claims of accuracy are about how well the model fit its training data or how accurate its predictions were post deployment – this is a critical point that AI companies may gloss over. Of course, incentive structures are aligned for them to do this.

Although Predictive AI exploded in use in the 2010s but there has been little published evidence of their efficacy and almost no company has decided to initiate an independent peer-reviewed test of its product’s effectiveness.

The authors share several examples of Predictive AI not working as advertised across domains such as:

• Law: COMPAS, ORAS, PSA
• Education: EAB Navigate
• HR: HireVue, Retorio
• Healthcare: Optum Impact Pro

In some of these cases the authors show that the accuracy of Predictive AI tools is not better than a coin flip (50:50). Please consider getting the book if you want to dive deeper into these specific examples.

When selling these tools, Predictive AI Snake Oil salesmen promise the glory of full automation, dramatically improved outcomes and significant cost reduction but then retreat to the fine print (such as, “there should always be a HITL checking the output of the AI tool”) when something goes wrong.

If you are in a position where you have to decide whether or not to go ahead with adopting a Predictive AI tool, it would be beneficial to ask for proof of real impact of the AI in a post deployment setting, instead of relying on claims of how well the model fit past data.

Incorrect proxies

Like I said, if your input (training data) does not really drive the output (prediction) then you will end up with spurious correlation and take incorrect decisions.

For instance, if an AI tool for hospitals wants to find out which patients are at most risk of disease to be able to offer preventive care, and uses amount spent on treatment over the past 12 months as a proxy then it may risk offering treatment to those who were already wealthy enough to afford treatment.

Bias

If the training data has bias, the Predictive AI will also be biased (“searching under the streetlight”) – for instance if the training data only/disproportionately has data from certain sections of society like the poor, marginalised, or people who use government aided services (such as medical aid) then it will single out those individuals in its predictions.

The authors say, “… predictive AI disproportionately harms … systematically excluded and disadvantaged [groups] …”.

For example, if you’re building AI to predict who will default on their loan, and the training data of past loan defaulters mostly included people will lower incomes, then you may have your loan application rejected if your income band is lower even though you were perfectly capable of paying back the loan.

When you build an AI on biased data then the AI just takes over that role of perpetrating that bias, and often without recourse to the affected parties.

Over-automation

As you’ll see in the case of Generative AI, an “automation bias” exists in Predictive AI use cases as well, but with far more serious consequences.

Over-automation refers to the situation when AI is used for decision making without offering any recourse to decision subjects.

The authors cite a few examples of how this has to financial loss such as the Australian Robodebt scandal. According to the authors, “… consequential decisions about people’s lives are being made using AI, and there is little or no recourse for flawed decisions.“.

“own impact”

Another key issue, and frankly a rookie mistake by the developers, of Predictive AI is that the model fails to take into account the impact of its own presence on the system.

For example, a Predictive AI tool designed to automatically identify students most in need for supplemental coaching based on their test scores may figure out that non-native English speakers do not need it, oblivious to the fact that the the manual system it is supposed to replace was proactively sending such students to the supplemental class and thus pre-empting a dip in their scores.

“AI can make good predictions if nothing else changes … it can’t tell us what would happen if something changed in a system …”.

Another case of “own impact” is when people try to game the system such as candidates wearing spectacles or adding books to the background when recording a video clip for a AI based screening tool.

The practical implication of this is that you should be asking the Predictive AI salesman on how he thinks people will game his product and what are the safeguards against that – not having answers for this will be a mark against the AI company.

Despite these shortcomings why is there so much hype around Predictive AI?

Part of it are the same reasons there is so much hype around AI in general – like I’ll cover below – it is the cool new technology today. Companies, researchers, journalists, public figures, society, all have a role to play in this.

Another key reason is that companies (and sometimes governments) get aboard Predictive AI because it allows them to reuse existing datasets.

But another, more hidden reason the authors link it to, is our inherent dislike for not being in control – by making predictions and claiming knowledge about what is going to happen is a way we feel that we have control.

That is why forecasting is such a spectator sport and has seen many players since the dawn of civilisation. Even outside AI there are political pundits predicting election outcomes, economist predicting the state of the economy at the end of the decade and so on.

The desire for predictability is evolutionarily programmed and whenever there will come a technology proclaiming to take away the randomness from your life, it will sell.

Generative AI, i.e. AI used to generate novel content based on past data, is all the rage these days since exploding on to the scene in 2022. And unlike its cousin, Predictive AI, it is better behaved.

Technically speaking, Generative AI is also “predictive” in nature. Like Predictive AI, it also makes predictions, but unlike Predictive AI, its predictions are about the same class of data as I explained above.

IMO, due to this fundamental right-alignment of the training and output data, and also of user expectations – incidents of snake oil in Generative AI are not as pronounced. The authors agree, “… the technology is powerful and the advances are real.“.

In the book, the Authors talk in depth about the history of ML, starting from the Perceptron, to ImageNet, to the role of GPUs and more. They also get into much detail about how Generative AI works including how individual neurons work, neural layers, vectors, deep learning etc. Please consider getting the book if you want to learn more.

So, in the case of Gen AI, it is less about snake oil and more about the issues arising from its use (or misuse).

The problem with Gen AI is not that it doesn’t work but that it works well enough to lead to two harms:

1. Everyone pressing, as Ethan Mollick calls it, “The Button” – This leads to a glut of artificially generated content with unintentional (or deliberate) consequences
2. People assuming that Gen AI can do much more than it actually can – This leads to over reliance on AI.

Let us talk more about these issues, and a few more, in the next section.

Bias

This is the big issue with Gen AI, and as I said in my Field Note – Co-Intelligence, is that its training data is frequently sourced from what AI developers—historically a relatively homogenous group, often male computer scientists from predominantly American and English-speaking backgrounds—could readily find, access, and perhaps assumed was free to use.

This very selection process, guided by their own inherent perspectives and limitations, can introduce a significant initial bias to what the AI learns about the world.

The problem of bias is then compounded by the fact that much more of the training comes from the open web, a space hardly renowned for being nontoxic, friendly or representative of balanced global perspectives.

“Dirty work”

After pre-training the AI, there is also the task of fine-tuning the AI, and this work is outsourced to low cost countries like Kenya where the workers are paid on average USD 2 per hour for sifting through graphic content and racist text. Apart from being poorly paid, the work lacks employment benefits and is unstable.

Credit or compensation

Crediting or compensating the creators of pre-training data for Gen AI is a grey area today. Gen AI is ideally supposed to be trained on copyright free content, but in reality it might not be. In such cases, ideally the creators of the content should be credited or compensated but they are not.

This problem is not a recent one as even back in 2009 when ImageNet was launched to the public it contained copyrighted images whose creators did not know their creations were being used as training data.

That precedent has carried on till date, and I feel if AI companies had to credit or compensate creators (say, in the way a Spotify compensates artists), they may quickly go out of business.

Even if you do not call yourself a “Creator”, consider this – say you are a long time contributor to websites like Quora or Reddit, helping the online community by offering your specific brand of expertise. Now, an AI tool like ChatGPT comes in and trains on your years worth of advice, and uses that to inform its responses. Since you were doing this work for free anyway, you may not want payment, but would you not want credit at least?

In this way, multi-billion dollar businesses have been built on a foundation dug by unpaid labour.

Companies are able to skirt the issue of creator compensation because of the way Gen AI trains – it only uses training data to refine its weights and does not plagiarise directly, and also that their use of training data falls under the “fair use” provision of copyright law.

Lately there has been a growing movement around artists demanding compensation for their content used for training AI, and now artists can now opt out of Generative AI. It will be interesting to see how the conversation will evolve in the coming years.

Over Reliance

There are many cases where people simply tend to blindly accept AI output (this problem also plagues Predictive AI) either out of laziness, or lack of critical thought, or lack of conceptual clarity, or even sometimes due to the hype. Ethan Mollick cites cases in his book, “Co-Intelligence”, of how even intelligent people are prone to essentially outsourcing their thinking to AI like the lawyer who used ChatGPT to conduct research for him only to later realise that ChatGPT had hallucinated. This is known as the “automation bias“.

Misuse

Because creating novel content is so easy (and getting easier) with Gen AI, people have gravitated to using it in unwise ways.

The incidents of misuse lie on a spectrum from students using it to write their essays to individuals using it to create sexualised images of real people (often celebrities) a.k.a. “deepfakes” to motivated parties using it to influence policy by creating a facade of public opinion to bad actors using it to rapidly generate misinformation to disturb peace.

The impact of this misuse, therefore, also ranges on a spectrum from people getting tired of AI generated content to completely suspending belief on anything they see on the Internet (this is called “liar’s dividend“).

Another facet of misuse is using AI for facial recognition.

Why is there hype around Gen and Predictive AI today?

The trigger was that Predictive AI and Generative AI both entered their latest AI springs in close succession (the former in 2010 and the latter in 2017).

And after the trigger, both forms of AI entered the Rube Goldberg machine that has companies, researchers, journalists and public figures pumping up the hype, eventually leading to customers buying the into it due to FOMO.

It works like this:

• A company releases a new AI product and hypes it up without telling how it actually works, making it feel like magic.
• News outlets (often underfunded) take the company’s PR claims at face value and publish.
• Then public figures (including company people) may try to get on the gravy train further pumping up the hype
• Which then gets many customers to buy the product despite no public, impartial evidence of product efficacy.

Let’s meet the players of the AI Hype Cycle:

The Companies

It doesn’t take a genius to realise companies have vested interest in generating hype around their products while lobbying against regulation that weakens their position and limited independent verification of claims behind the veil of “trade secret” – so they will. AI continuing that tradition is nothing to be surprised about.

Companies may game metrics to project improved accuracy of AI tools, they may cherry pick examples of tasks (in a job) where the AI does well and hype that up as proof that the whole job can be automated away, they may tout performance on benchmark metrics while being silent about real world utility. The authors show examples in the book.

The Researchers

The academic community is a weak check on the AI industry because many experts choose to accept industry positions instead of joining academia.

When it comes to AI research there is lesser focus on scientific understanding of AI tools as long as they work well – this lends an unwarranted aura to AI.

The authors also talk about the reproducibility crisis in AI research where independent peers are unable to reproduce the claims made of AI capabilities.

The issue is not helped by the fact that most AI research papers “… satisfied merely twenty to thirty percent of the reproducibility requirements … making it hard to even investigate if the results were reproducible.“.

The News Media

News media, the fourth pillar of a strong democracy, has been under a lot of financial strain recently due to the “… rise of social media and click-driven journalism [leading to] decrease in the ability to do in-depth reporting profitably.“.

Everyone plays the “growth game”, even news outlets, and sensationalism is not new in journalism nor is it unique to AI, it’s just unfortunate that AI, already shrouded in mystique, becomes further enigmatic as news outlets chase their clicks: whether its oversimplifying nuance, uncritically repeating PR statements, fanning popular fears (such as a AI takeover), overusing images of robots, attributing agency to AI, relying on grand metaphors, engaging in “access journalism” – none of this helps make an informed public.

Public Figures

Public figures are supposed to be the torch bearers dispelling the hype and fear around new technologies while wisely using their power and influence. Nowhere is this more important, perhaps, than in AI today.

Unfortunately, certain public figures in AI have added to the hype than dispelling it – the authors give the example of Schmidt, Kissinger and Huttenlocher and the book they wrote, “The Age of AI” (which I also reviewed here on Sunchaser). They say that the “… book is littered with AI hype.“, not very different from my own comment that the book seemed like an “… opinion piece than a factual and researched representation.“.

When we describe AI (or anything really) in enigmatic terms – whether we are praising it (“It’s like magic!”) or criticising it (“It’s voodoo!”) – we unconsciously reduce the feeling of agency we feel over the situation, and this problem is compounded when people in positions of power and influence do the same thing.

Feeling a lack of agency makes us passive even during situations where we actually stand a change to make a difference.

In accidentally portraying AI as something that is enigmatic instead of something that we can (and actually do) understand reasonably well, public figures add to the AI Hype Cycle.

And some public figures may even have a dismissive attitude towards work of other domains prompting them to make sweeping statements about AI’s potential – such as Geoffrey Hinton who told us in 2016 that, “People should stop training radiologists now” because within five years AI would have replaced them.

The Customers

While the parties above may be supplying the AI hype, it is the “broken institutions” that are buying it, and if the buying will stop, the selling will too.

These broken institutions are underfunded and understaffed public schools, hospitals, clinics, welfare providers, law and order departments, or even local governments.

These institutes are groaning under workload while more work keeps on piling up (like insurance providers), or they are trying to stay up to date with a world that is rapidly changing (like schools), or they are trying to be proactive and prevent accidents (like the police) – some may also be facing all these issue at once. And as is the case with people, so is the case with organisations – when you’re desperate “… AI can seem like a silver bullet, even if it is snake oil.“.

“… dubious AI is disproportionately adopted by institutions that are underfunded or cannot effectively perform their roles. These are the institutions we call broken.“.

If you work for such an institution then you can do your bit by spreading education around how AI (esp. Predictive AI) works and its limitations – prompting decision makers to think again about seeing AI as a silver bullet.

Social media has many problems today – hate speech, misinformation, cyber bullying, online scams to name a few. That is why content moderation is so important. If it weren’t for content moderation, many of us would get turned off with all the unmoderated content.

Social media platforms toe a fine line between content recommendations and content moderation, showing you what will make you stay on the platform while just avoiding that which would irk you (or be illegal).

Traditionally, content moderation has been done by humans but is trauma inducing work that has significant negative effect on the moderator’s mental health. Given that AI has become so good at identifying and tagging pictures, and reading and interpreting text – one would think this is an area where AI would actually be fully useful without any snake oil.

Yes and no. In some areas of Content Moderation AI does help today, but the scale today is limited – restricted to only the posts in blatant violation of policy. For the more subtler stuff (and there is a lot of it) human intervention is needed. AI tools tend to interpret the content of social posts too literally today and are unable to understand the context as well – for instance it may remove a post about a chess move if it contains the phrase “white is better”. The authors expect this to improve in the coming years.

But there are some areas where content moderation is just hard, even for humans – in these areas AI is not expected to improve things. Consider our ever evolving language (who’d have thought that the eggplant and peach emojis would be considered indicators of sexualised text), or when the content policy itself changes due to social movement.

There is also the issue that content moderation rules today, whether for AI or for humans, suffer from delayed application, are inconsistent even when applied, pander to the government in power, are designed for the world but with mostly U.S./western sensibilities in mind and often end up over-blocking/under-blocking content. The authors spend time discussing this and several other topics like the many incidents of violence that social media played a part in (and that the platforms could have prevented), copyright law and its lopsided implementation, role of oversight boards, crisis of trust, how the problem of problematic content is really a self-made one by the platforms, how they’re optimising for the wrong thing with “engagement” and much more at length in the book – though these topics are outside the goal for this Field Note (and hence not covered), they make for enlightening reading.

If you are looking for “AI Snake Oil” to tell you that AGI (Artificial General Intelligence) is the biggest case of snake oil then you’re out of luck. The authors show that computing is moving towards increasing generality and that AGI may be likely achieved one day.

But that doesn’t mean we’re all doomed and AGI is coming this Thursday. And they are completely dismissive (with reason) of AGI doomsday scenarios.

First, it’s not that the road to AGI is easy – often we see that if one problem with AI is solved then another emerges. The authors give the example of how only when we had figured out how to get ML to work did we realise the importance of collecting large datasets to feed into the ML model, before having working ML we did not even know that the dataset problem existed.

Second, we simply do not have enough past data for anything like AGI to be making predictions about when it will come or what it will look like. According to the authors, “… probability forecasts [for AGI] are simply guesses dressed up with the veneer of mathematical precision.“.

Thinking narrowly or unliaterally about AGI, that is, “AI will come and the world will just sit around and do nothing” also is symptomaic of the “hedgehog” approach (as popularised by Philip Tetlock). But in reality it is the “foxes”, i.e. people who integrate information from many domains before forming their worldview, that fare better when it comes to forecasting the future.

Third, AGI (at least initially) will have to operate in a world of humans, which is much different from operating in its self contained digital environment. When it starts to operate in this world, say as an anthropomorphic robot or maybe even the AI running a paperclip factory (😉), it will have to navigate our systems in a way that we do not get skeptical of its intentions. If any AI is stupid enough to take seriously the paperclip maximiser hypothesis then it will be quickly shut down. Whereas any AI powerful enough to actually deliver on the nightmarish end of the paperclip maximisation goal will also be smart enough to know not take it literally.

So, it’s not that AGI is not coming or it’s not possible, but pinning a date on it or being doomsday-ish about is pointless. Instead of AGI alone, we focus on what motivated individuals can do with it (or even with AI today). Per the authors, the need is to defend against specific threats of AI being used by humans.

Towards the end, the book starts to read more like a note on the popular brand of capitalism in power through most of the world and its natural impact on AI.

Although capitalism technically is a free-for-all with only the invisible hand deciding who will survive and who will perish (and indeed has proven to be the more sustainable and generally beneficial approach in the last five decades), in practice when a player gets too large it may lead to competition getting stifled.

Lack of competition combined with private ownership is monopolistic. And that is why, when it comes to certain essential goods and services, governments like to keep the competition going – either by regulation or by entering into the field themselves.

Is AI important enough to be classified as “essential”? If it is, then recent corporate developments seem to be taking this essential service “… from public knowledge to trade secrets.“.

The authors make it clear that this “privatisation” of AI is a bad thing – it will be interesting to see how much of a private playground corporate lobbyists make AI in the coming years.

Closely related to the principles of capitalism is “optimisation“, like I said, the going understanding is that the invisible hand will always lead to a more optimal allocation of resources in a capitalistic structure. And sure, it might.

But this has also led us to believe that optimisation should be the end goal everywhere, and so when AI shows up promising hyper-optimisation, we get all excited. Let’s talk about this.

All successful people are lottery winners. If you are reading this, and have made it thus far, you are likely one of them – most people do not have the luxury of time to read (or indeed, life circumstances that would have allowed them to even appreciate) content like this.

Do you consider yourself all self-made? Who gave you your intelligence? Or your height? Or your good looks? Did you snatch them out of fate’s hands?

We are all born into very different circumstances that off the bat define our life trajectory, and that is before taking into account the randomness of luck as we grow.

Consider a hypothetical example: a child is born into a destitute family, but luckily with a psychology that makes him work harder and longer than his peers, over the years through his hard work he lifts himself out of poverty and starts working at a local grocery store, one day while going to work he is hit by a truck, this accident paralyses him below the waist but also makes him a savant, in ten years he goes on to cure cancer and wins the Nobel prize, and in fifteen his company has made him a billionaire.

Would you say that this man is self-made? Is all of his success his own?

Having pride in overcoming obstacles to arrive at your current station while being grateful that luck was also on your side is very different from being pompous about your victories and dismissing the role of luck.

Realise that there is nothing called a meritocratic society – we ignore the role of randomness and luck while selecting candidates on the basis of their scores because it is more expedient to feel that we’re in control of our destiny.

As the authors agree, beyond a a baseline level of skill for each “bracket” of success, it starts getting random.

How does this link with optimisation?

In believing that there is one “best” that can be arrived at – in this context through the application of AI – in every situation in life, even the incredible complex ones, we merely set ourselves for failure and ironically, perhaps sub-optimal outcomes.

Letting go of this compulsive need for optimisation (perhaps more endemic to MBAs like myself than anyone else) will make us more resistant to AI snake oil and its peddlers that claim to extract the last drop of “efficiency” from your system. As a solution, the authors make a case for “partial lotteries” and show how it can make outcomes more fair.

Given how powerful (and increasingly widely used) AI is, regulation seems essential. Yet, the popular sentiment around AI regulation is that it should be avoided because:

1. We think regulation will stifle AI development.
2. We think that AI is a new beast that needs entirely unique regulatory frameworks.
3. We think regulators would be incapable of regulating (or even understanding) something as dynamic and complex as AI. And that self-regulation is the only option.

Regulation stifling growth

While it is true that overzealous regulation can limit development of a technology, this does not mean that it will necessarily do so.

In fact, given that AI companies are driven by profit, regulation is important to limit unchecked pursuit of profit and place guardrails cordoning off the forbidden areas such as training AI on copyrighted content without notifying or compensating creators. “When companies have no incentive to address the harms brought about by their business, regulation is essential.“.

Unique frameworks

The authors demonstrate how the frameworks needed to govern AI are already in place in many countries and give examples of the vertical approach in the U.S., the horizontal approach in the EU (GDPR, DSA, DMA, AIA), and also China’s first-vertical-then-horizontal approach.

Incapable regulators

While it is true that regulators may not be subject matter experts (SMEs) in every field they regulate, but they don’t need to be – because they can work with the actual SMEs from a field and accordingly understand the current situation, expected trajectory and impact, in this way they can draft regulation that lays down the foundational principles of how thinking is to be informed for a given field.

The claim that only AI companies can self-govern is also summarily dismissed – AI is not a mystery and its fundamental working can be taught to intelligent people, and more importantly, AI companies are for-profit entities. Asking them to self-regulate may lead to perverse incentives.

The authors give the example of how when Sam Altman testified before the U.S. Senate, he called for regulation drafted favourably towards OpenAI.

This is not surprising or new to Big Tech, we’ve seen this before from Big Tobacco to Big Pharma.

If regulators continue to think of AI as an enigmatic being they risk regulatory capture which is not good news for anyone (except perhaps the few AI companies).

In pursuing a deeper understanding about AI, a few things are clear IMO when it comes to our future with AI:

1. Sellers will continue to tout AI as a panacea but eventually people will realise some of it is snake oil.
2. AI will not automate jobs, but rather tasks. Jobs, which are a bundle of tasks, will continue to evolve.
3. The power balance between labour and owners of capital as powerful AI technology will be seen as a form of capital, as a result labour unions will be more important.
4. Generative AI is going to be significant creative enabler.
5. The value of human made content will increase with the Generative AI content glut.
6. AI will lose its sheen and just become “normal” technology.

These points bore out in my read of “AI Snake Oil”, and also “Co-Intelligence“.

Time will tell what actually happens, but the authors share their advice:

• Generally, the awe and mystique around AI needs to be reduced by getting people to understand broadly how it works and what it can/cannot do – a world where there is a “… heightened sensitivity around automated systems that make life-changing decisions about people, and broad awareness about their dangers.“.
• Experimentation with and responsible use of AI by the general public -whether in teaching, knowledge work or others – needs to be encouraged but at the same time the knowledge that AI is an evolving technology (and not a panacea) will need to be imparted.
• AI regulation will be important but it should not be from a fear of the unknown (and hence risk shutting down innovation) but instead a proactive approach that encourages companies to innovate but in that innovation also enable independent third parties to verify their claims.
• Increasing independent third party evaluation of AI products and claims will also need to be systemically ensured by “… increasing funding for enforcement agencies, research, and investigative journalism.“.
• Interoperability between social media platforms as well as the ability to tailor one’s social feed to support their personal development goals using Content Moderation AI.
• Laws around creator credit and compensation and AI companies should not be allowed to profit unilaterally at their expense.
• Government support for lifelong learning and perhaps UBI and other safety nets to limit the negative consequences of displacement of certain jobs by AI.