Are You Blocking the Crawlers That Produce Your AI Citations?
OpenAI updated the documentation for its crawlers this year, and the changes are the kind that quietly break assumptions people wrote down two years ago and never revisited.
The headline changes: OAI-SearchBot is no longer described as feeding the navigational links inside ChatGPT answers — so blocking it does not mean you will not appear in those links. ChatGPT-User is now described as also handling Custom GPT requests and GPT Actions, and the robots.txt wording was narrowed so that the compliance statement refers specifically to OAI-SearchBot and GPTBot. And OAI-SearchBot and GPTBot now share information between each other: if your site allows both, one crawl may serve both use cases.
There is one more line worth remembering, because it reframes the whole exercise: for most sites, OpenAI crawls far more than it ever sends back as traffic.
Why this matters beyond the model vendor
Most teams never made a deliberate decision about AI crawlers. They inherited one. Somewhere between a CDN template, a security plugin’s default rules, and a line someone added to robots.txt during a bad week of bot traffic, an accident got baked in.
The result is a class of site that is fully readable by traditional search engines and partially or entirely invisible to the crawlers that assemble AI answers. Nothing announces this. There is no error, no alert, and no visible difference in your analytics — because the failure mode is the absence of a citation you never received.
A fifteen-minute check
| Check | Where to look | What to fix |
|---|---|---|
| robots.txt rules for AI agents | Your robots.txt | Remove broad User-agent: * disallows |
| WAF or bot-management rules | CDN or host dashboard | Whitelist verified AI crawlers |
| Uptime and rate-limit rules | Server logs | Ensure limits don’t 429 the crawlers |
| Blocked paths for API or app routes | Robots and WAF | Keep private paths blocked, open public pages |
| Response codes for logged-out visitors | Fetch a key page anonymously | Fix anything gated that should be public |
The last row catches the most embarrassing failures, because it involves no bot configuration at all. Pages that require a JavaScript challenge, a cookie consent gate, or a login to render any content are effectively invisible to crawlers that do not run a full browser, no matter what robots.txt says.
The asymmetry worth internalizing
If a vendor crawls you far more than it cites or refers you, then your server logs are not a demand signal. They measure how interesting your site is to a system that is trying to answer questions about your category — not how much traffic you are owed.
That asymmetry has a practical consequence for submission work. Every link you place, every profile, every listing is an instruction to read a page. If the destination page cannot be fetched by the crawlers that matter, the placement is doing half its job — and no submission service can fix that for you. What https://backlinksubmit.org covers is the other end: matching relevant platforms to your product, submitting manually, and showing in a console which platforms have been submitted and how each one is progressing — with about 90% of placements indexed, and anything a directory never indexes resubmitted or replaced.
Checking that your own pages answer a crawler is thirty seconds of work, and it is yours.
What to decide deliberately
Blocking AI crawlers is a legitimate choice, and plenty of publishers have made it for training-related agents. What is not legitimate is doing it by accident, and the difference costs you nothing to establish:
Decide about training, then decide about answering. These are different actions with different consequences. Many sites are comfortable being read in order to answer a question and uncomfortable being used for model training. Vendors have separated the user agents to allow exactly that distinction.
Open the pages you want cited. Your homepage, pricing, product, comparison and support pages should be readable by any agent that can fetch HTML. If a page exists to inform a buyer, gating it from a crawler only removes you from the answer.
Re-test after every platform change. CDN migrations, new bot-management add-ons and security plugins are the three most common ways a working configuration silently stops working. Put an anonymous fetch of your three most important pages on a quarterly checklist.
FAQ
Does blocking OpenAI’s crawlers stop me appearing in ChatGPT answers?
Not necessarily, and the rules changed. OAI-SearchBot is no longer described as feeding the navigational links in answers, so blocking it does not by itself remove you from those links.
Which user agents should I allow?
That is a policy decision about training versus answering. Read the current vendor documentation rather than a blog post from two years ago — the descriptions are updated periodically, which is how this year’s changes appeared.
Do AI crawlers respect robots.txt?
OAI-SearchBot and GPTBot are documented as respecting robots.txt; the wording narrowed this year to those two agents specifically. Check the current documentation rather than relying on an older summary.
How do I know whether my site is being read?
Compare crawler hits in your logs against citations and referrals. If crawlers are frequent and citations are absent, the problem is usually the material available for them to cite, not access.
Share Article
