Report That Cloudflare AI Bot Blocking Prevents Googlebot From Indexing Sites

There’s a report that blocking AI coaching bots with Cloudflare can also be blocking respectable Googlebot and Bingbot from indexing an internet site. Cloudflare not too long ago introduced that this may occur starting in September. It’s unclear whether or not this can be a fluke or consumer error but it surely does function a reminder to concentrate on the pending change and to regulate settings accordingly.

Redditor Claims Cloudflare Blocked Googlebot

A Redditor within the r/web optimization group requested if anybody else has skilled Cloudflare blocking Googlebot as an AI coach.

They asked:

“Has anybody else run into Cloudflare AI Crawlers blocking Googlebot/Bingbot?
I’m testing Cloudflare’s new AI Crawlers & Scrapers characteristic and seen one thing sudden.

Once I set AI Coaching = Block, each Googlebot and Bingbot begin receiving HTTP 403 responses when attempting to fetch my sitemap. As quickly as I disable the AI Coaching block, the sitemap is accessible once more.

From what I perceive, Cloudflare now classifies Googlebot and Bingbot as “Search + Coaching” bots, so enabling the coaching block seems to have an effect on them as effectively.

This leaves me with a dilemma:

• Permit AI Coaching so search engines like google can entry my sitemap.
• Block AI Coaching and threat Google/Bing getting 403s.

Has anybody discovered a clear workaround?”

Google’s John Mueller responded to ask for a DM to take a better look. One other Redditor requested if perhaps Cloudflare was blocking rogue bots that had been spoofing Googlebot’s consumer agent.

The unique poster replied that wasn’t the case and supplied extra info:

“I perceive that Cloudflare blocks pretend Googlebots, however in my case I’m referring to what Cloudflare itself exhibits within the dashboard.

Once I allow Bot Combat Mode, importing my sitemap returns a 403 error. As quickly as I disable it, the 403 disappears.

Additionally, within the AI Crawlers part, Cloudflare exhibits Googlebot and Bingbot as blocked robotically. That’s why I’m questioning if Bot Combat Mode is interfering with respectable crawlers or if the dashboard is reporting one thing else.”

Cloudflare’s Official Documentation

In it’s official announcement states that starting September 15 Cloudflare will begin blocking mixed-usage bots, that are bots like Googlebot that concurrently index web sites for search and acquire information AI coaching. The impact is that Cloudflare will block Googlebot.

Cloudflare’s announcement explains:

“On September 15, 2026, Cloudflare will set up to date defaults for brand new domains: bots labeled as Coaching or as Agent will likely be blocked on pages that show adverts, and Search will stay allowed.

Blended-purpose crawlers that mix Search and Coaching may also be blocked by all configurations to dam AI coaching, together with the legacy “Block AI bots” choice. Earlier than September 15, all prospects can choose out of those new defaults.”

That type of blocking is meant to enter impact in September so it’s curious that somebody is reporting that that is occurring proper now.

Featured Picture by Shutterstock/Erlin Diah