A report updated on October 1 looked at 4,242 robots.txt files parsed by Cloudflare at the end of September. I expected it to say websites are blocking AI harder than ever. It said something more useful. In the authors' words, the big change over the past year "is not more blocking, it is more explicit allowing."
Site owners have started treating AI crawlers as two groups. GPTBot, which collects pages to train OpenAI's models, is named in 520 block rules and 326 allow rules. OAI-SearchBot, which fetches pages so ChatGPT can cite them in an answer, runs the other way: 295 allows against 268 blocks.
I think that split is correct. Whether your content trains a model is a policy question. Whether an assistant can read your auto loan page when a member asks about auto loans is a visibility question. One small file answers both, and most credit unions I look at have never been asked either one.
Where it goes wrong
Two places.
The first is the blunt block. A separate study in September checked the 5,000 most visited sites on the web. Of the ones blocking GPTBot, 44.5 percent also block OAI-SearchBot. Some of them meant to. Plenty copied a "block AI" snippet and took themselves out of ChatGPT's citations without ever deciding to.
The second is the one marketing never sees. The same Cloudflare data shows 13.47 percent of AI bot requests last quarter were answered with a 403, which is a flat refusal. A year earlier it was 5.98 percent. That refusal does not come from robots.txt. It comes from the firewall or bot protection that sits in front of the site, usually a setting someone in IT or at the web vendor switched on. Your robots file can say welcome while the front door says no.
What I saw this month
I am working with a financial institution that is getting ready to relaunch its website. Before launch I read the staging build one page at a time, then requested a sample of pages while identifying as each major crawler: Google, Bing, OpenAI, Anthropic and Perplexity. Every one of them got through. Good build, good vendor.
The robots file was the loose end. It was a staging file, never meant for the public, with a sitemap line that pointed at a developer's machine. There is nothing wrong with that on a staging site. The risk was in the fix. One reasonable-looking edit would have told every crawler to stay out of the entire site on launch day. And nobody would have caught it for weeks, because the site would have looked perfect to every person who visited it.
That is why I treat this as a launch checklist item and not an SEO footnote.
What to check this week
Open yourdomain.org/robots.txt in a browser. It is plain text and you can read it without a developer. Look for three things. A line that says "Disallow: /" under "User-agent: *" blocks everyone. Any group that names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended is a decision somebody made, so find out who made it and when. And the Sitemap line should point to your live domain.
Then ask whoever runs your firewall or CDN one question: are we challenging or blocking AI bots, and which ones? Cloudflare and its competitors ship one-click settings for this now. If the answer is "I'm not sure," write that down. That is the finding.
Then make the two decisions on purpose. My default for a credit union is to allow the answering bots, because a member asking an assistant about your rates should get your rates from your page and not from an aggregator's. Training is a judgment call for your leadership and compliance team, and I can argue it either way. What I cannot defend is not knowing.
If a relaunch is coming, put robots.txt on the launch-day list by name. Read the live file within an hour of the switch. Request a few pages as each crawler and confirm they come back clean. The whole thing takes about fifteen minutes.
A redesign gets months of attention on how the site looks to people. The file that decides how it looks to machines is a few lines of plain text. Somebody should read it.
If you want a second set of eyes on yours, request the free scan. Put "check our robots file" in the notes field and I will include crawler access in what I send back.