robots.nxt blog

Bots are half your traffic. Now that you can see them with robots.nxt, here's how to control bots on your website.

Hiding or ignoring bot traffic is no longer an option. Half of internet traffic is bots, and only a few are ones you want on your website. Here's how to see it.

You have to do a million things, layered on top of each other and interlocking, brick by brick by brick, each in the right place at the right time.

Internet content is the raw material that AI relies on, and AI will pay for it if we make them.

AI can give an answer, but not a solution, because there’s no human on the other side to provide what someone's looking for.

“They’ll pay for it because AI companies don’t have a business if they don’t have content to train on,” I told him. “Content is the raw material for AI.”

Ads and subscriptions don’t work for this new AI-centered bot-populated internet. That's why we built robots.nxt.

The landscape of the internet has shifted, creating a blind spot for bot traffic on websites. You can't ignore bots anymore.

Welcome to the dawn of a new age of the internet, web3. Every age brings a new monetization strategy. For web3, that strategy is monetizing bot traffic.

If we can't change other people's minds — most of the time — why do we let other people change the way we feel?

Sales are an important but often ignored step in building a startup.

Measure your progress from where you started, not where you're going, because the horizon moves, but your starting point is fixed.

Startups put a lot of emphasis on launch day, but most startups fail to build an audience and need multiple attempts before they get anyone's attention.

The internet was built for humans, of humans, by humans. But humans are no longer the largest audience on the internet.

AI companies live and die by their access to training data for their models. To get that content, they do what has been a common practice since the dawn of the internet - they crawl publicly available websites.

I had the idea, I had enough skills, and I had a version that was at least somewhat functional. But I couldn’t keep making headway by myself. Time is the eternal enemy of the mortal. It was time to bring in reinforcements.

Most people pick a spot in life and decide they know everything they need to, and they're not going to learn anymore. That attitude is death to an entrepreneur.

A handful of observations about the state of the internet showed me what I should work on next.

A new start requires a new approach and a new mindset. This time I wanted to try to match Silicon Valley startup standards instead of doing it my way.