The short answer
In the seven days to October 12, 2026, Cloudflare recorded 1,432,401 verified requests from AI crawlers and AI search bots across 40 websites that Cheap Cheap Websites builds and runs. In the same week, verified search engine crawlers made 88,063 requests. That is about 16 AI requests for every one from a traditional search crawler.
AI systems are now the most active readers of small business websites. Whether they can read your pages cleanly, and whether those pages answer real questions, decides whether your business can show up when someone asks ChatGPT, Claude, Perplexity or an AI search box for a recommendation.
What we measured
We pulled request analytics for every website in our network from Cloudflare, the security and delivery service that sits in front of them. Cloudflare checks bots against their published IP ranges and signatures and puts verified bots into categories. We used those verified categories for the headline numbers, because anyone can pretend to be "GPTBot" in a user-agent string, and many scanners do.
The 40 websites include a news publication, a large local business directory, Spanish- and English-language finance guides, industry websites for restaurants and contractors, state business websites and this website. Together they publish about 478,000 indexable pages, about 74,000 of them written pages and the rest directory listings.
| Verified category (Cloudflare) | What it is | Requests, 7 days |
|---|---|---|
| AI Crawler | Bots that collect pages to train or ground AI models | 916,684 |
| AI Search | Bots that fetch pages for AI search answers | 515,717 |
| Search Engine Crawler | Traditional search indexing such as Google and Bing | 88,063 |
| Search Engine Optimization | SEO tool crawlers | 62,662 |
| Page Preview | Link previews in chat and social apps | 44,168 |
| AI Assistant | An assistant fetching a page live because a person asked it something | 1,599 |
Finding 1: AI bots read far more than Google does
For years, "can Google crawl it" was the only technical question that mattered for a small website. In our data, Googlebot made about 64,000 successful requests in the week. AI crawlers and AI search bots made more than twenty times that.
The biggest readers by volume were AI search and answer engines, followed by the crawlers run by large social, shopping and device companies. The crawlers that feed ChatGPT and Claude were smaller in volume but steady every day. Volume is not the same as value: one live fetch by an assistant answering a customer matters more than thousands of background crawls. But a page that has never been crawled cannot be quoted at all.
Finding 2: Live assistant fetches are rare, which makes each one count
The "AI Assistant" category is the closest thing to a customer in the logs. It is recorded when an assistant such as ChatGPT or Claude opens a page in real time because a person asked it a question. Across 40 websites this happened 1,599 times in a week, a little over 200 a day.
Those fetches go to pages that already answer a specific question well: a cost page, a "how do I" guide, a comparison. The assistant is checking a source it already found. That is why we write every page around one real question and answer it in the first paragraph.
Finding 3: A lot of bot requests never get a page
About a third of all bot requests we measured ended without a normal page. The biggest causes were rate limits on very large sites, links to pages that no longer exist, redirects, and bots that were blocked by security rules.
On a small business website the same problems show up at a smaller scale: an old page that now returns "not found", a redirect chain left over from a redesign, or a security setting that blocks AI bots by accident. We found and fixed exactly that on our own network in 2026: a firewall rule that was challenging AI crawlers on our sites. After the fix, AI crawlers could read the sites again.
| Response | Share of bot requests measured |
|---|---|
| Page served (2xx) | 67% |
| Blocked, missing or rate-limited (4xx) | 30% |
| Redirect (3xx) | 3% |
| Server error (5xx) | under 0.1% |
Finding 4: Content-rich sites get read the most
Reads were not spread evenly. The two websites with the most published material, the directory and the news publication, took about four in five successful AI reads. Websites with a few hundred pages were read thousands of times a week. Websites with fewer than 60 pages were read far less.
For a local business, the lesson is not to publish 400,000 pages. It is that every useful page is another thing an AI system can read, understand and quote. A five-page website gives an assistant five chances. A website that also answers the 30 questions customers actually ask gives it 35.
What this means for your website
- Do not block AI crawlers by accident. Check your security settings and your robots.txt file. Our free robots and sitemap checker shows what your site tells bots.
- Give bots a map. An XML sitemap and an llms.txt file tell crawlers which pages matter.
- Fix dead pages and redirect chains. Every "not found" is a wasted read.
- Answer real questions, one per page. Assistants fetch pages that answer what the person asked. See our AI search basics guide.
- Add structured data. Schema markup tells machines your business name, location, hours and services without guessing.
Every Cheap Cheap Websites build ships with all five: a clean sitemap, an llms.txt file, schema, AI crawlers allowed, and pages written around real customer questions.
Limits of this study
This is one week of data from websites we own, not a sample of the whole internet. Our network has more pages than a typical small business website, so the totals are larger than you would see on a five-page site. The share of AI traffic, the failure causes and the pattern of which pages get fetched are the parts that carry over.
Cloudflare's verified categories rely on Cloudflare's own bot verification. Requests from bots that fake a famous name are not counted in the verified totals. Request counts are machine reads, not human visitors, and they do not show how often an AI system actually mentioned a business in an answer.
Sources and method
- Cloudflare request analytics for 40 websites we own, 7 days to October 12, 2026, using Cloudflare's verified-bot categories.
- Cloudflare, "Verified bots" documentation (developers.cloudflare.com/bots/concepts/bot/verified-bots/).
- Response-code shares are computed from the named crawlers we tracked individually (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, bingbot, Applebot, Amazonbot, meta-externalagent, Bytespider, DuckAssistBot, MistralAI-User).
- Websites are not named. You may quote these figures with a link to this page.
