Beta

See which AI agents take your data,
and decide what they can do

Stilebase shows you every AI agent on your website or API, tells the real ones from the impostors, and lets you set the rules for each one. People and search engines are never touched.

Robots.txt only asks.
Not every AI agent listens.

Some AI agents say who they are. Others borrow a crawler’s name, hide behind a browser or run as plain scripts. And a blanket block also shuts out the AI search that sends you visitors.

The web is already closing the door

We checked more than 11,000 websites, including over 6,000 listings, jobs, cars, travel and price-comparison sites. About a quarter already keep AI agents out.

24%
of sites already block AI agents
~40%
of the world’s 10,000 most visited websites
32%
of car sites and classifieds
1 in 6
top websites tell AI crawlers to stay out

Already telling the major AI crawlers to stay out

Each blocks GPTBot, ClaudeBot and PerplexityBot in its own robots.txt

  • Amazon
  • The New York Times
  • CNN
  • LinkedIn
  • X
  • Pinterest
  • REUTERSReuters
  • WSJThe Wall Street Journal
  • IMDb

Logos are trademarks of their owners, shown to illustrate their public robots.txt policies. No affiliation or endorsement.

Bigger sites block more often

Sites grouped by popularity rank (#1 = most visited site on the web)

Site sizeSites checkedBlock AI agents
Very large (top 10K)5,947
39%
Large (10K–50K)513
39%
Mid-size (50K–250K)3,018
29%
Smaller (250K–500K)4,166
21%
Small (500K–1M)1,897
19%

Blocking is a yes-or-no switch: the agents you’d welcome leave, and the scrapers that get through take your data anyway. Stilebase adds a third option: know who each agent is, and decide for each one.

Stilebase scans, October 2026: 11,741 domains, mostly listings, jobs, travel and price sites (9,784 answered), plus all of the 10,000 most visited domains: 5,947 websites once CDN, ad and API hosts are excluded, counting sites that ban all unknown bots while letting search engines in (36% counting only the 5,683 sites we could open). Named sites: their own robots.txt blocks GPTBot, ClaudeBot and PerplexityBot, as checked 13 Oct 2026. “Block” means bot protection on content pages or AI crawlers disallowed in robots.txt. Category figures: 1,308 car sites and 103 classifieds sites. Popularity rank from the Tranco list; the top-10K row covers all 5,947 websites.

Not every “GPTBot” is GPTBot

Scrapers put AI crawler names in their user agent to get past allowlists. A real GPTBot comes from the IP ranges OpenAI publishes, but most sites never check. Find out who’s really taking your data.

Who's really visiting

Every AI agent that came, and whether it's verified or only borrowing a name, checked against the IP ranges and signatures the AI companies publish.

What they took

Pages and bandwidth per agent, and what serving that traffic costs you.

What to do about it

Which agents are worth letting in, and which to stop.

Get your free AI-traffic report

Create a free account and send us one day of your access logs. We’ll reply with your report. OpenAI’s published crawler IPs ↗

How It Works

Three steps. Nothing changes for your visitors.

01

Add your site

Add your domain, then drop in our Cloudflare Worker or a few lines of code. Your service itself doesn't change.

02

See every agent

Each AI agent's request is checked against what the AI companies publish, so you can tell the real ones from the impostors in your dashboard.

03

Set your rules

Let in the agents that bring you visitors, and stop the rest. Change your mind at any time, agent by agent.

Know who’s really there

Stilebase tells you who each agent is, and lets you decide what it can do.

Verified agents

Every agent is checked against the IP ranges and signatures AI companies publish. A scraper calling itself GPTBot is caught, so are agents hiding behind a browser and scripts like curl, and scripts posing as a browser can be made to prove it.

Your rules

Free passes for agents that bring you visitors, like AI search. Allow only verified agents. Block or allow any agent or company.

Every visit on record

What each agent asked for, how its request identified it, and whether that checked out.

Safe by design

People and search engines are never touched. If Stilebase is unreachable, your site works as usual. We check your integration every few hours.

Built for sites with data worth taking

Listings, jobs, cars, travel and price comparison: live data that AI agents come back for again and again.

For site and API owners

  • One Cloudflare Worker or a few lines of code
  • Real agents told apart from impostors
  • Free passes for agents that send you visitors
  • Allow only verified agents if you like; block anyone
  • Every agent visit in your dashboard

The Crawler Tax

AI scrapers hit your API for free, and you pay AWS for every byte they take. See what that costs you.

stilebase ~ crawler-tax --region us-east-1

$ configure --traffic

1K1M
0.1 MB10 MB

Assumes AWS data transfer out at $0.09/GB and a 30-day month. 1 GB = 1,024 MB.

$ crawler-tax --report

egress/day   292.97 GB
egress/month 8,789.06 GB
calls/month  4,500,000
Current AWS Loss
-$791.02/mo
Egress you pay to serve scrapers for free.

See who’s taking your data

Create a free account, add your site and the integration, and see the AI agents visiting it.