AI companies scrape huge amounts of data from the internet.
They feed that data into a large language model, which produces a matrix of probabilities for how people put words together — if this, then that.
Those probabilities describe how likely a unit of data — a “token”, is to appear after another token in a huge chain, a probabilistically weighted markov chain.
The AI feeds the prompt through its “parameters” — the weights of probabilities that its generated from its data set that shows how likely one word is to follow another — to produce the most probable string of tokens to follow your prompt. Then it spits out that string of tokens as the answer.
You can get poor quality discussion of essentially any topic just by ingesting reddit, youtube comments, twitter, and so on. Across 2024 AI companies struck licensing deals with these large “user generated content” sites.
AI companies like Meta have ingested 82 terabytes of written works, so they can tell you about books. They’ve been licensing research databases to include high-knowledge content.
But suppose you ask an AI about your tooth extraction?
How to fix your roof?
What kind of attorney you need?
What kind of therapy can help with your trauma?
It needs those high-minded academic or professional topics to be translated out of technical language and into the way average people actually talk and write, and point to a person near you who knows how to do that for you.
You’re not going to get that from reddit, twitter, or youtube comments.
You’re not going to get that from a book, or an academic paper.
You’re going to get that from content marketing websites.
Professionals translate their knowledge into plain language guidance on topics that people need to know a little bit about, so they can hire the right professional to deal with it correctly.
These high-value professional services have been producing content marketing materials that explain what services they provide and why in plain language so that when a non-professional searches for a complex topic, they can get an easy understanding about what it means.
People search for information about the surgical procedure their general practitioner recommended and find a content marketing post explaining the topic. Then they fill out an inbound lead form — “Contact Us!” — and the professional gets back to them and makes an appointment.
This was all working beautifully, until AI companies scraped the content and fed it into a training system. Now the AI company can speak knowledgeably about the professional topic to answer someone’s question.
AI gives an answer…
But not a solution, because there’s no professional to provide the service.
AI overviews means click through rates on content marketing posts have collapsed.
At the top of the lead generation funnel is content marketing. Content includes keywords for search terms that a business provides. Search engine optimization (SEO) improves search engine ranking (SER) for those keywords.
The better content ranks in search, the more likely someone who searches sees the link. The better the answer to the question, the more likely they click the link. That’s called click through rate (CTR).
Once they’re on the landing page, they’re at the top of the funnel. Then they fill out an inbound lead generation form (“Contact Us!”) so the business has a lead. That lead goes into a client relationship management (CRM) system and someone follows up.
That’s how modern professional services work.
AI overviews destroy that model.
The AI reads your content, and spits it out to answer the search question. AI overviews reduce click-through-rates by >80% (4% CTR without AIO to 0.64% CTR with AIO).
That means if your answer is good enough that the AI uses it to respond to someone’s question, you don’t get anything out of it.
The sales opportunity never reaches a landing page, they don’t even know who the answer came from. The prospect never fills out a lead form, they never reach a CRM.
You don’t have a brand, you don’t have credibility, you don’t have a lead.
You were good enough for the AI answer, and the reward is that it killed your funnel.
Soon you don’t have a business. At least, not business from web leads.
You know who does reach your landing page and fill out a form? A bot.
A bot steals your content. They steal your click throughs. They steal your leads. And to add insult to injury, they flood your CRM with dick pills and “transaction management” scams.
AI content harvesting is rapidly destroying the very environment that makes AI possible. It’s a snake that eats its own tail. AI content scraping is killing the very environment that provided the content that AI relies on to train itself.
What happens next? Where do we go from here?
What happens to Google when they can’t serve ads because nobody clicks through to pages? What happens when businesses stop buying ads because they get nothing from it? What happens when businesses stop producing content that AI relies on to train?
The internet will die, just like a forest dies when we pollute its water and air.
AI content harvesting is a beautiful example of a negative externality. Like how a factory gets an economic benefit from polluting the air or water, AI gets an economic benefit from harvesting content for free from the web to train.
But in the same way that, eventually, air and water pollution harm the community that the factory depends on to run it, sell, and buy its products, AI content harvesting creates a negative externality that destroys the environment that makes AI possible — rich, abundant content.
As far as I know, economics gives us solutions to externalities in pricing and conditionals.
Content is the raw material for AI. Just like you can’t make steel without iron, and can’t make a car without steel, you can’t make AI without content, and you can’t make content without human knowledge and effort.
We have to price content and impose conditions on its use.
If the people producing content put a price on it, and force AI to pay to use the content, with terms and conditions, we can eliminate that externality and support the creators that AI relies on. We need AI to attribute where it gets its knowledge, so it can refer people to the source.
And we need management and controls on our websites so that we can impose these prices and conditions.
This is why we’ve built robots.nxt, so that we can harness AI for the good of the internet, so that content creators can get paid, get attributed, get business, and thrive.
If we don’t act now to defend ourselves and our creative energy, AI will turn the internet into a post-apocalyptic Terminator style wasteland.
And what will you do then?
Talk to someone in real life?
Interact with the people who live near you?
Go outside again?
Press X to doubt.
So let’s do something about it.