AI Crawlers Are Scaling Up for the Shopping Season. Check Your WAF Rules.
Someone watches bot traffic for a living, and they noticed OpenAI adding a lot of new IP ranges ahead of Thanksgiving and Black Friday — described as a series of /28 blocks, with a “significant ramp up over the past one to two months” that does not include the ranges OpenAI already had.
The advice attached to that observation was practical and brief: make sure these are added to your WAF rules, or use your CDN’s bot detection instead.
What to expect in your logs
The mechanism is straightforward. Shopping season means more people asking AI assistants about products, which means more pages being fetched to answer them. If your site is in a category people shop in, your crawler traffic is going up, and it will go up unevenly — a real spike around the peak days, not a smooth curve.
Three things follow, and only one of them is a problem.
Request volume is not a demand signal. OpenAI crawls far more than it refers. A log full of AI bot hits tells you that you are an interesting source about your category, not that traffic is coming.
Capacity is shared. The same server limit serves Googlebot, Bingbot and every AI crawler. A crawler spike during your busiest commercial week is a resource event, which is worth knowing before it happens rather than during it.
The real risk is your own defences. A WAF rule that is too broad, a bot-management profile that blocks unrecognised ranges, or a rate limit tuned to human traffic will start blocking new IP ranges the moment they appear. That is how a site ends up unreadable by the systems it wants to be cited by, during the exact month when being cited is worth most.
A pre-season checklist
| Check | Why before the peak |
|---|---|
| Known AI crawler user agents allowed | New ranges arrive under documented agents |
| WAF rules reviewed for broad CIDR blocks | Custom blocks don’t know about new ranges |
| Rate limits separated by traffic type | Humans and crawlers should not share a bucket |
| Log monitoring with per-agent breakdown | You cannot see a spike you are not tracking |
| Public pages reachable without JS or login | Fetching tools do not run a browser |
The second row is where most accidental blocking lives. Custom blocklists age badly: they are written once, during a bad week of bot traffic, and nobody revisits them when a legitimate crawler adds forty new ranges.
The counter-argument, and why it is a decision rather than an accident
There is a legitimate position that says: block them all, we get nothing back. Plenty of publishers have taken it, and for training-related agents it is a defensible policy.
What is not defensible is arriving there by accident, and the two are easy to confuse because the failure mode looks identical from inside your analytics. The difference is whether someone decided, wrote it down, and can explain why.
If you do allow them, the standard is simple: public pages should be fetchable by anything that can request HTML, and private or expensive routes should be blocked deliberately rather than by an inherited rule.
Where this sits relative to the rest of the work
Crawler access is the precondition, not the strategy. Being readable gets you considered; it does not get you described. That still comes from pages about you existing in places that have their own audience.
The two halves are usually owned by different people and reviewed at different times. This is the mechanical half of the second one: on https://backlinksubmit.org you enter a product URL, it matches relevant directories and communities from a handpicked list, the team submits manually, and a console shows which platforms have been submitted and how each is progressing. Access checks belong in the same quarterly review, ten minutes away from the list.
FAQ
Does more AI crawling mean more AI traffic?
No. Crawling measures how useful your site is as a source, not how many visitors you are owed. The gap is usually very wide.
What should I add to my WAF rules?
Allow the documented AI crawler user agents rather than hard-coding IP ranges, which change. If your CDN offers verified-bot detection, use it instead of maintaining a list.
Will a spike break my site?
Generally not from volume alone, but it will consume shared capacity in your busiest week. Watch per-agent request rates and keep crawler traffic in a separate rate-limit bucket from human traffic.
Is blocking AI crawlers a reasonable choice?
As a considered policy, yes — particularly for training-related agents. As an accident caused by a stale blocklist, it is usually discovered at the worst possible moment.
Share Article
