On 6 August one of the sites I run served 1,153 search impressions. On 7 August it served 119. By 12 August it was down to 16. Average position moved from 10.8 to 67.9 — from the first page of Google to the seventh.
Nothing had been deployed. The site was up the whole time, responding in under a second. Every page returned HTTP 200 in a browser. Search Console's URL Inspection said, for every URL I tested, Submitted and indexed. The pages were not removed, not deindexed, not penalised.
They were simply no longer being crawled, and the reason turned out to be a single toggle in Cloudflare that had been on for a while and had never caused a problem before.
This is the full diagnostic trail: what the data looked like, why the obvious explanations were wrong, the one header that settled it, and the control experiment that removed the last doubt.
What does a crawl failure look like in Search Console?
It looks like a ranking collapse, not an indexing error — which is exactly why it is easy to misdiagnose. Here is the daily series across the break:
| Date | Impressions | Clicks | Avg. position |
|---|---|---|---|
| 4 Aug | 1,550 | 16 | 10.0 |
| 5 Aug | 1,666 | 8 | 10.7 |
| 6 Aug | 1,153 | 16 | 10.8 |
| 7 Aug | 119 | 0 | 67.9 |
| 8 Aug | 50 | 0 | 54.4 |
| 12 Aug | 16 | 0 | 51.5 |
The device split made it look even stranger. Mobile went from 550 impressions on 6 August to 3 on 7 August. Desktop fell from 598 to 116. Tablet disappeared from the report entirely.
A drop like this has a small number of plausible causes, and I worked through them in order.
A deploy broke something. No deploy had happened. Git log was quiet for days on either side of 7 August.
The pages got deindexed. They did not. I ran URL Inspection against ten URLs — the homepage, several detail pages, a category page. Every one came back PASS / Submitted and indexed, with rich results detected. If Google had dropped them, this is where it would show.
A manual action or algorithmic penalty. A penalty does not usually take a site from position 10 to position 68 across every page type in one day and leave the index intact.
robots.txt or a stray noindex. robots.txt was clean and permissive — I keep explicit Allow blocks for the AI crawlers in there. No page carried a noindex in its source.
The thing that actually pointed at the answer was hiding in the URL Inspection output, in a field I nearly skipped: last crawl date. The most recent crawl across every URL I checked was 7 August. It was 12 August when I looked. Googlebot had not fetched a single page in five days, and the crawl had stopped on exactly the day the rankings fell.
Why was the index fine while the rankings were gone?
Because indexing and crawling fail independently. Google kept serving the pages it already had, but everything that depends on continued access — freshness, re-evaluation, the ability to confirm the page still matches the query — decayed. The URLs stayed in the index and slid down it.
That is why the shape of this failure is so misleading. Every tool that reports on index status says the site is healthy. The failure is in access, and almost nothing in Search Console reports on access directly.
What the request actually returned
I stopped looking at dashboards and made requests. Fetching a page over curl returned this:
$ curl -sSI https://example.com/some-page
HTTP/2 403
cf-mitigated: challenge
server: cloudflare
server-timing: chlray;desc="a2a170726c3335c3"
cf-mitigated: challenge is Cloudflare stating, in the response itself, that it intercepted the request and served a challenge instead of the origin's content. The body confirmed it — the familiar interstitial:
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
<meta name="robots" content="noindex,nofollow">
There it is. The challenge page carries noindex,nofollow.
That single line explains the entire ranking collapse. A crawler that receives this page does not see your article. It sees a short HTML document with no content, no links, and an explicit instruction not to index or follow. Serve that to Googlebot repeatedly and you are telling Google, in the strongest markup available, that the URL should not be in search results.
The behaviour was not consistent, which had masked it for days. The same URL returned real content on one request and a challenge on the next. It is rate- and reputation-driven, not deterministic. A human clicking around the site would very likely never see it.
Which Cloudflare setting causes this?
Bot Fight Mode — a single toggle under Security → Settings → Bot traffic, described in the dashboard as "Detect and challenges bot traffic on your domain."
The critical detail is what it does not do. Bot Fight Mode is the free-plan feature, and it does not carve out an exception for verified crawlers the way Super Bot Fight Mode does on paid plans. It classifies traffic as automated and challenges it, and a search engine crawler is, by every technical definition, automated traffic.
Security Events confirmed the mechanism from Cloudflare's side, with Managed Challenge as the action and Bot fight mode as the service.
There is a second-order effect worth knowing about. Because the challenge is what gets served, every non-browser client is affected, not just search crawlers. In my case an SEO health check flagged ClaudeBot: HTTP 403 days before the rankings moved. I read it as an AI-crawler access question and moved on. It was the same failure, reported early, in a channel I was not treating as urgent.
How do you prove it was the CDN and not something else?
Run a control experiment. This is the part I would repeat on any infrastructure diagnosis, and it happened to be free here.
The affected site is not this one. It is a finance dashboard I run separately, and it happens to share everything with the site you are reading: same server, same Traefik instance, same Cloudflare account, same free plan, same DNS and certificate setup. Different Cloudflare zone.
theinfinity.dev — this site — was completely healthy throughout. Its crawl was current, its rankings were untouched. That made it an unusually clean control.
So I compared the two zones setting by setting and found exactly one meaningful difference: Bot Fight Mode was on for the broken site and off for the healthy one.
| Broken site | Healthy site | |
|---|---|---|
| Bot Fight Mode | On | Off |
| Challenge rate (47 requests) | 3 of 3 sampled | 0 of 47 |
cf-mitigated header | challenge | absent |
| Googlebot last crawl | 5 days stale | current |
| Ranking impact | position 10 → 68 | none |
Same host, same proxy, same plan, one variable. That is as close to a controlled experiment as production infrastructure normally allows, and it is worth remembering that running two properties in one account gives you this for free.
One measurement did not work, and it is instructive. I filtered Cloudflare's Security Events to Google's ASN (15169) expecting to find a pile of challenged Googlebot requests. There were none. For a moment that looked like it exonerated Bot Fight Mode.
It did not — it confirmed the diagnosis. There were no challenge events from Google because Googlebot had stopped coming. Crawlers back off from hosts that stop returning content. By the time I went looking for the evidence, the traffic that would have produced it had already dried up. An empty log is not the same as a clean log.
What the fix looked like
Turning Bot Fight Mode off, and replacing it with something targeted.
The reason it was on in the first place is real: one IP in Hong Kong was responsible for 1,080 requests in 24 hours, roughly a third of all traffic to the site. That is a scraper and it deserves a response — just not one that catches search engines in the same net.
A custom WAF rule does the same job with a scalpel:
(ip.src eq 203.0.113.10 and not cf.client.bot)
Action: Block. The not cf.client.bot clause is deliberate — if that address ever becomes a verified crawler, the rule stops applying to it rather than silently blocking a search engine.
Verification, immediately after:
# 24 requests across 4 URLs
temiz=24 challenge=0
$ curl -sSI https://example.com/some-page | grep -E 'HTTP|cf-mitigated'
HTTP/2 200
# cf-mitigated header is gone
And a bot sweep, since the whole point was crawler access:
| Crawler | Before | After |
|---|---|---|
| Googlebot | challenge | 200 |
| Bingbot | challenge | 200 |
| ClaudeBot | 403 | 200 |
| GPTBot | challenge | 200 |
The last step was resubmitting the sitemap to nudge the crawl back. Recovery is not instant — the rankings have to be re-earned as Google re-crawls — but the metric to watch is not position. It is the last crawl date in URL Inspection. When that starts moving forward again, the fix has landed.
How do you tell if this is happening to you?
Four checks, in order of how quickly they give you an answer.
Look at the response headers, not the page. curl -sSI https://yoursite.com/ and look for cf-mitigated. A browser will not show you this because a browser solves the challenge transparently. If the header is there, you have your answer in one request.
Check the last crawl date, not the index status. In Search Console, open URL Inspection on a few pages and read the crawl date rather than the verdict. If it is stale by days while your content is current, crawling has stopped even though indexing looks fine.
Repeat your requests. Challenges are probabilistic. I had URLs return real content and a challenge on consecutive requests. A single clean curl proves nothing — send five.
Watch the boring bot in your monitoring. The ClaudeBot 403 in my SEO check was the earliest signal available and it preceded the ranking loss by days. Non-browser clients are the canaries here, because they hit the challenge long before anything shows up in a search dashboard.
FAQ
Does Cloudflare Bot Fight Mode block Googlebot?
It can. Bot Fight Mode is the free-plan feature and it does not exempt verified crawlers the way Super Bot Fight Mode does on paid plans. When it challenges a request from a search crawler, the crawler receives an interstitial page carrying noindex,nofollow instead of your content.
Why do my pages still show as indexed if Googlebot is blocked?
Indexing and crawling fail independently. Google continues to serve pages it has already indexed, so URL Inspection keeps reporting Submitted and indexed. The signal that reveals the problem is the last crawl date, which stops advancing.
What does the cf-mitigated header mean?
cf-mitigated: challenge means Cloudflare intercepted the request and returned a challenge page rather than passing it to your origin. It appears alongside HTTP 403. It is the fastest single piece of evidence that a CDN rule, not your application, is responsible for what a client is seeing.
Will turning off Bot Fight Mode expose my site to scrapers?
It removes one blunt layer, so replace it rather than simply deleting it. A WAF custom rule targeting the specific address, ASN or behaviour you actually want to stop does the same work without catching search engines. Adding not cf.client.bot to the expression keeps verified crawlers out of the rule permanently.
How long does it take for rankings to recover?
There is no fixed number, because recovery depends on Google re-crawling the affected URLs and that happens on its own schedule. Watch the last crawl date rather than position — crawling resumes first, and rankings follow it.
Can I keep Bot Fight Mode and still allow Googlebot?
Not reliably on the free plan, because the exemption for verified bots is a paid-plan capability. The practical options are to turn it off and write targeted rules, or to upgrade to a plan where Super Bot Fight Mode lets you allow verified crawlers explicitly.
Does this affect AI crawlers too?
Yes, and usually first. The challenge is served to any client that fails the bot check, so GPTBot, ClaudeBot and PerplexityBot hit it the same way Googlebot does. If you have deliberately allowed AI crawlers in robots.txt, a CDN-level challenge silently overrides that intent.
What to check now
If you run anything behind Cloudflare on a free plan, this takes about a minute. Send curl -sSI at your own site five times and look for cf-mitigated. Open Search Console and read the last crawl date on three URLs. Then open Security → Settings and look at whether Bot Fight Mode is on.
The failure mode here is not that something broke loudly. It is that a protective setting did exactly what it advertised — challenge automated traffic — and search crawlers are automated traffic. The site stayed up, the pages stayed indexed, the dashboards stayed green, and the traffic left anyway.
If you want the measurement discipline that surfaced this, the same approach shows up in our cache eviction benchmark: stop reading about the behaviour and measure it, and prefer the check that can only come out one way.