Evolutionary Call and Response
What’s he building in there?
AI companies live and die by their access to training data for their models.
To get that content, they do what has been a common practice since the dawn of the internet - they crawl publicly available websites.
That’s never been a problem before, historically it’s largely welcome. That’s how search engines index sites so they can promote search results. That’s how search engine optimization works, so that sites can rank higher in results.
Crawling has traditionally been embraced by websites, and gently managed through a file called robots.txt which politely requests what parts of the site the bots should visit and what sites they should avoid.
But something has been happening quietly in the background. The sea level of the internet has changed, but our websites haven’t, and now those shining cities are drowning.
The internet was built for and by humans. Humans have always been the primary audience online, and our analytics, our management tools, and our monetization methods are intended for humans.
We identify our audiences, we build content, applications, and services to motivate and serve those audiences, and we find ways to generate cash flow from their visits.
But humans are no longer the largest audience.
Bots are the largest audience online
Like most changes, it happened very slowly, then fast. This is a classic S-curve.
You see this everywhere with technology. Here’s another example, a user adoption curve. This describes how persons adopt new technologies.
All human products are a form of technology, even things we consider simple, like string or a button, were at one time an incredible innovation. And over time, as we have gotten more industrial, our rate of adoption has only increased.
See how vertical those more recent adoption curves are?
For years, bots were a small minority of internet traffic, mostly beneficent, intended to help understand the structure of the internet.
Malicious bots emerged, bot nets, bot farms, performing DDOS attacks, cracking websites, spreading viruses. Cybersecurity evolved, content delivery networks were formed, to help prevent and mitigate this new cohort of harmful bots.
But AI has created a new category of autonomous agents - commercial operators. They are neither beneficent nor malicious, they are merely serving the purpose of their corporate masters. And that purpose is explicitly commercial, or to be more blunt, financial.
A hit dog hollers
The first to cry out about this problem were digital publishers. They’d taken it from every side for decades.
The transition from paper to digital killed their subscriber networks. They lost control over their own ads, as external corporations like Google built independent ad networks better, faster, cheaper than newspapers could.
Craigslist took want ads. Dating sites & apps snuck off with personals.
Bit by bit technology companies consumed publishers, stealing away all of their revenue sources and leaving only their cost center - content creation - naked and exposed.
Content aggregation sites like Reddit, Twitter, and Facebook killed the on-page browsing model where someone would “read the paper” and go through each new story every day in turn. Instead of news from one source - the local paper, the national magazine - people got fragments from a thousand places around the world.
Publishers put up paywalls, and people stopped showing up. Publishers demanded pay-per-click from aggregators for supplying the content that people obtained through the aggregator. That… worked, a bit, sort of, but too little, too late.
People kept sharing but stopped clicking - the headline, and maybe the lede, already populated into the aggregator, were enough for many. They “got the gist” and moved on. No click revenues, no paywall registrations.
Publishers have resorted to increasingly more egregious tactics - clickbait, ragebait, pandering, ambiguous headlines, buried ledes - to get people to visit their site so they can make some money.
The Lone and Level Sands Stretch Far Away
The final insult - the tide turned, and publishers were flooded with traffic: the wrong traffic. As the percentage of human traffic is diminishing, bot traffic is increasing. But bots don’t generate revenue. In fact, AI bots effectively preempt revenue.
AI companies send hordes of webcrawlers to scrape content, every single tiny bit of content, from the entire internet, analyzing it with billion-dollar AI models, and selling access to the final product, the AI model.
The final injury - for many, there’s no reason to visit the original site at all, because the AI model can provide a comprehensive summary of the topic, no click required, and with all the reading and thinking pre-performed by something else.
I met a traveller from an antique land,
Who said—“Two vast and trunkless legs of stone
Stand in the desert. . . . Near them, on the sand,
Half sunk a shattered visage lies, whose frown,
And wrinkled lip, and sneer of cold command,
Tell that its sculptor well those passions read
Which yet survive, stamped on these lifeless things,
The hand that mocked them, and the heart that fed;
And on the pedestal, these words appear:
My name is Ozymandias, King of Kings;
Look on my Works, ye Mighty, and despair!
Nothing beside remains. Round the decay
Of that colossal Wreck, boundless and bare
The lone and level sands stretch far away.
It all just kind of happened over night!
But as Lenin once said,
"There are decades where nothing happens; and there are weeks where decades happen"
COVID held many of those weeks for us, where the entire world changed over night, every night.
Most of us woke up every day for years at a time still thinking it was yesterday, but yesterday came suddenly.
There is Always a Solution
The ground shifted overnight. The sea level rose. And again, for the nth time in a few short decades, publishers were drowining.
There had to be a life preserver we could throw them.
And, as in most problems in life, the cause of the problem provides for its own solution - from the right perspective.
What perspective will solve this crisis?