See which AI agents take your data,
and decide what they can do
Stilebase shows you every AI agent on your website or API, tells the real ones from the impostors, and lets you set the rules for each one. People and search engines are never touched.
Robots.txt only asks.
Not every AI agent listens.
Some AI agents say who they are. Others borrow a crawler’s name, hide behind a browser or run as plain scripts. And a blanket block also shuts out the AI search that sends you visitors.
The web is already closing the door
We checked more than 11,000 websites, including over 6,000 listings, jobs, cars, travel and price-comparison sites. About a quarter already keep AI agents out.
Already telling the major AI crawlers to stay out
Each blocks GPTBot, ClaudeBot and PerplexityBot in its own robots.txt
- Amazon
- The New York Times
- CNN
- X
- REUTERSReuters
- WSJThe Wall Street Journal
- IMDb
Logos are trademarks of their owners, shown to illustrate their public robots.txt policies. No affiliation or endorsement.
Bigger sites block more often
Sites grouped by popularity rank (#1 = most visited site on the web)
| Site size | Sites checked | Block AI agents |
|---|---|---|
| Very large (top 10K) | 5,947 | 39% |
| Large (10K–50K) | 513 | 39% |
| Mid-size (50K–250K) | 3,018 | 29% |
| Smaller (250K–500K) | 4,166 | 21% |
| Small (500K–1M) | 1,897 | 19% |
Blocking is a yes-or-no switch: the agents you’d welcome leave, and the scrapers that get through take your data anyway. Stilebase adds a third option: know who each agent is, and decide for each one.
Stilebase scans, October 2026: 11,741 domains, mostly listings, jobs, travel and price sites (9,784 answered), plus all of the 10,000 most visited domains: 5,947 websites once CDN, ad and API hosts are excluded, counting sites that ban all unknown bots while letting search engines in (36% counting only the 5,683 sites we could open). Named sites: their own robots.txt blocks GPTBot, ClaudeBot and PerplexityBot, as checked 13 Oct 2026. “Block” means bot protection on content pages or AI crawlers disallowed in robots.txt. Category figures: 1,308 car sites and 103 classifieds sites. Popularity rank from the Tranco list; the top-10K row covers all 5,947 websites.
Not every “GPTBot” is GPTBot
Scrapers put AI crawler names in their user agent to get past allowlists. A real GPTBot comes from the IP ranges OpenAI publishes, but most sites never check. Find out who’s really taking your data.
Who's really visiting
Every AI agent that came, and whether it's verified or only borrowing a name, checked against the IP ranges and signatures the AI companies publish.
What they took
Pages and bandwidth per agent, and what serving that traffic costs you.
What to do about it
Which agents are worth letting in, and which to stop.
Create a free account and send us one day of your access logs. We’ll reply with your report. OpenAI’s published crawler IPs ↗
How It Works
Three steps. Nothing changes for your visitors.
Add your site
Add your domain, then drop in our Cloudflare Worker or a few lines of code. Your service itself doesn't change.
See every agent
Each AI agent's request is checked against what the AI companies publish, so you can tell the real ones from the impostors in your dashboard.
Set your rules
Let in the agents that bring you visitors, and stop the rest. Change your mind at any time, agent by agent.
Know who’s really there
Stilebase tells you who each agent is, and lets you decide what it can do.
Verified agents
Every agent is checked against the IP ranges and signatures AI companies publish. A scraper calling itself GPTBot is caught, so are agents hiding behind a browser and scripts like curl, and scripts posing as a browser can be made to prove it.
Your rules
Free passes for agents that bring you visitors, like AI search. Allow only verified agents. Block or allow any agent or company.
Every visit on record
What each agent asked for, how its request identified it, and whether that checked out.
Safe by design
People and search engines are never touched. If Stilebase is unreachable, your site works as usual. We check your integration every few hours.
Built for sites with data worth taking
Listings, jobs, cars, travel and price comparison: live data that AI agents come back for again and again.
For site and API owners
- One Cloudflare Worker or a few lines of code
- Real agents told apart from impostors
- Free passes for agents that send you visitors
- Allow only verified agents if you like; block anyone
- Every agent visit in your dashboard
The Crawler Tax
AI scrapers hit your API for free, and you pay AWS for every byte they take. See what that costs you.
$ configure --traffic
Assumes AWS data transfer out at $0.09/GB and a 30-day month. 1 GB = 1,024 MB.
$ crawler-tax --report
See who’s taking your data
Create a free account, add your site and the integration, and see the AI agents visiting it.