Notes · · 2 min
Two weeks of work on AI visibility, and Cloudflare was turning the bots away at the door
I wrote service pages and added schema so ChatGPT and Perplexity could read the site. Then I measured; every AI bot was getting 403. The cause was not the content, it was a rule enabled with one click.
At the end of August I set a goal for a small business site I maintain: when someone asks ChatGPT who builds WhatsApp assistants for clinics in İzmir, we should be one of the first answers. For two weeks I wrote three service pages, added Service and FAQ schema, filled in the Google Business Profile, registered the site in Search Console.
On measurement day I looked at the Cloudflare dashboard first. Under security rules:
AI Crawl Control - Block AI bots by User Agent · Active
The rule blocked 16 bots on every path except robots.txt, including ChatGPT-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot and bingbot. In the previous 24 hours, 163 of 212 AI crawler requests had been rejected. Allowed requests for OpenAI, Anthropic, Perplexity and Bing: zero.
How I measured
I did not trust the dashboard; I tested from outside, requesting the same page with different User-Agent strings:
Googlebot 200
ChatGPT-User 403
PerplexityBot 403
bingbot 403
Two weeks of writing had gone into pages the bots could not read.
Where the rule came from
Cloudflare's AI Crawl Control screen generates this rule from a single Block button. It was probably pressed months earlier with the reflex "AI should not scrape my content". Reasonable that day; today the same button turns away the bot that would bring the customer.
There was a second layer: the managed "Block AI bots" setting under Security settings. Even after narrowing the custom rule, the search bots kept getting 403 until that was switched off too.
What I did
I did not delete the rule; I narrowed it. Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) stay blocked. Search and live-fetch agents (ChatGPT-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot) are allowed. Then the same test again:
ChatGPT-User 200
OAI-SearchBot 200
PerplexityBot 200
bingbot 200
GPTBot 403
ClaudeBot 403
The lesson
Check the door before writing content. If your site is behind Cloudflare, look once at Security → Rules and Security → Settings → Bot traffic. Anything with "AI" in its name may be blocking the search bots too.
I will measure again a week later. I do not know when the bots will come back; if the first numbers are low, that goes here too.
If you think I measured something wrong, write to me; I will correct it and note the change here. Contact →