Deep Questions Deserve Deep Answers
On New Years Eve, a woman came up to me and asked a very deep question.
“What did it all cost you?” she said, looking me up and down.
I was taken aback. What a thing to ask a stranger.
“What do you mean?”
It turned out she was referring to my coat.
“I got it from my dad,” I told her, “I don’t know what it cost him.”
That’s true in any number of ways, isn’t it?
A deep question deserves a deep answer.
When I was sitting in the debris of my last company, I knew I wanted to do something fundamental. So I started asking myself deep questions.
I like to take big swings. I’d rather work on something interesting than something easy. Maybe that’s a character flaw.
Maybe that’s just who I am.
What I wanted to do this time was pure software, internet tools, to help with the transition from the old human-centric internet to the new AI-centric internet dominated by automated agents - bots.
I realized that ads and subscriptions didn’t work for this new internet population. Bots aren’t driven by impulse, whimsy, curiousity, or intrigue like humans are.
Bots do what they’re told.
But what have these bots been told to do?
I always tell people, you have to ask the right questions to get the right answers.
Someone you’ve never met before shows up at your door. What’re the right questions?
Why are you here? What do you want?
The first question is, What do these bots want?
And the answer is, These bots want to scrape your content to feed into an LLM.
Either someone scrapes a bunch of websites and puts their archive onto a marketplace where an AI companies buys it to use for training, or the AI company scrapes the content from a website directly.
The AI training market was around $3b in 2024, and AI en-masse companies started licensing content from major websites to try to stop the flood of lawsuits.
But there’s 2 billion websites, and most aren’t big enough to properly threaten a lawsuit or negotiate a licensing deal. They need something else.
The next question is, How do we make this scale?
And the answer is, They need a software solution.
A Theory of Business Enterprise
One of my former employees, an economist by education, once gifted me a fascinating book called “The Theory of Business Enterprise” by Thorstein Veblen.
At first I was taken aback by Veblen’s theories, but the more I thought about them, the more sense they made.
One theory Veblen espoused that I found particularly offensive on first blush was that a businessman’s primary objective was not to enable business. On the contrary, the businessman’s primary objective is to prevent business from occuring unless, and until, the businessman is assured of being paid.
For Veblen, the businessman does not get paid by enabling business, just the opposite, the businessman prevents business from being done unless he gets paid.
This is a fascinating insight.
Take that concept and apply it to any business and you will soon find it true.
McDonalds’ doesn’t enable people to get fed, McDonalds’ prevents people from getting fed unless McDonalds’ gets paid.
United Healthcare doesn’t enable people to get healthcare, it prevents people from getting healthcare unless United Healthcare gets paid (and often not even then).
Whether this is brilliant or cruel or some combination of both isn’t the point, the point is that it’s true, and I can’t change it.
When something is true and beyond my power to change, then I have no choice but to either avoid it or wield it as a tool. Since I have no option but to engage in business, I am left with no option but to wield that truth as a tool.
I realized that we could embody Veblen’s theory by building a system doesn’t enable content scraping from websites, per se, but a system that prevents content scraping from occuring unless the website operator gets paid.
It’s Ladies Night for the Internet
Bars tend to have a problem — they get a lot of dudes looking for women, and not nearly as many women looking for dudes. They invented a very simple solution — women get in free, guys pay a cover charge.
What a great idea!
We used that as a concept model for our product. Websites want human traffic, because that makes them money. Websites either don’t care about, or don’t want, most bot traffic, because that costs them money.
We built robots.nxt so that it can tell the difference between human and bot traffic. Human traffic passes through freely, but bot traffic has rules applied to it.
Those rules actively enforce where bots can go, and what bots can do. That’s why it’s called robots.nxt, a play on robots.txt, which is a file most websites have that politely requests bots to behave.
But robots.nxt isn’t asking nicely, like robots.txt, we’re forcing. Bots cannot break the rules you set in robots.nxt.
And those rules let you turn on a bot paywall for any portion of your website so that the only way the bot can access that page, that route, that file, that whatever, is if the bot pays you whatever price you set.
robots.nxt prevents bots from accessing your website unless you get paid.
Welcome to Pay-Per-View Internet for Bots
It’s very simple. If the bot pays the price, robots.nxt serves them the page, the route, the content, the file the bot requested. If the bot doesn’t pay, they get ignored.
Let’s go back to my inciting questions: What’s the point of it all? Why do we do this? Who cares?
The point is to make sure that websites can generate revenue from their (new) largest audience — bots.
2/3 of those bots are malicious, and won’t pay, so we can just block them. Letting them in is like a bar full of people who not only don’t drink, but are actively vandalizing the bar and starting fights. Why tolerate that? Get rid of ’em. You just saved the cost of serving pages to all that wasted traffic.
1/3 of those bots are commercial, scraping content to use for AI training, or to sell to someone else to use for AI training. That’s like people stealing bottles to sell out the backdoor — it’s costing you money and hurting your market reach.
Some of those bots you want in for free, because they’re doing something useful — indexing or whatever. They’re workers that are helping you draw in other traffic.
Some of those commercial bots aren’t helping and won’t pay, so throw ’em out and save the money.
And most importantly, some of those commercial bots will convert to paying, because they have no choice — if they can’t get content, whoever runs the bot doesn’t have a business. A content market for AI, or an AI company directly.
Your content is the raw material for those businesses. Why would you give it away?
Someone is going to make money from your content! Why not you?
The person who steals your catalytic converter and sells it at the scrapyard — oh, sorry, I mean — scrapes your content and sells it in the marketplace, and the AI company both have a commercial motive.
PaynPoint wants the first person that makes money from your content to be you.
With robots.nxt installed, the website operators, content creators, and copyright owners get paid by bots who want to access their content.
That way whoever hosts, created, or owns the content gets a benefit from the commercial use of their content.
Content creators get exploited by platforms all over the internet — on TikTok, on Youtube, on Amazon, on Spotify, on Facebook, and a million other places large and small. These are ad-supported or subscription-driven platforms. The platform steps in the middle between the market and the content creator and takes the lion’s share of the benefit.
With robots.nxt, we hope to level the playing field, and give content creators a new way to generate revenue directly from their own website, where they have direct control over the pricing.
At the same time, robots.nxt makes it possible for AI companies to get content direct from the source, direct from the website, without middle-men or legal liabilities.
Because robots.nxt works with any website, and sets up in just a few minutes with barely any technical skill, we’d like to think this democratizes content markets and gives creators independence from 3rd party controlled platforms.
That’s why we do it, that’s what we’re here for.
We hope we can help you.
Set up robots.nxt on your website today, it’ll only take about 5 minutes.