Cloudflare SEO Settings That Help or Hurt Crawling

Cloudflare sits between your visitors, search engine crawlers and your server. That position is useful, because it can add HTTPS, filter attacks and speed up delivery. It is also why a single dashboard setting can change what Googlebot and Bingbot receive. This guide goes through the Cloudflare SEO settings that matter for crawling, explains what each one does, and shows how to check that it is not getting in the way.

There is no "SEO switch" in Cloudflare, and nothing here is a ranking trick. The goal is smaller: make sure the security and speed features you turned on are not returning challenges, redirect loops or stale pages to the crawlers you want, while still doing the job you turned them on for.

The guide is for site owners who already use Cloudflare, or are about to move their DNS there. By the end you will have a short audit list, a test for each setting and a clear idea of which settings to leave alone. If a crawler is already being blocked, start with Cloudflare Blocking Googlebot or Bingbot: How to Diagnose, then come back here.

Before you begin

You need dashboard access to the zone (your domain in Cloudflare), the ability to run curl from a terminal, and Google Search Console access if you have it. Note down the current value of every setting before you change it, so that you can undo a change. Most settings take effect quickly, which also means a mistake reaches visitors quickly. Make one change at a time and test after each.

Step-by-step instructions

Step 1: Choose the right SSL/TLS encryption mode

The SSL/TLS encryption mode controls how Cloudflare talks to your origin server. In Flexible mode Cloudflare connects to the origin over HTTP. If your origin also redirects all HTTP requests to HTTPS, the two keep redirecting each other until the browser reports ERR_TOO_MANY_REDIRECTS. Crawlers meet the same loop and cannot fetch the page.

Cloudflare's own fixes are to remove the origin's HTTPS redirect, or to switch to Full or Full (strict), which require an SSL certificate on the origin. Related loops come from "Always Use HTTPS" combined with an origin that redirects back to HTTP, from an origin that redirects HTTPS to HTTP in Full modes, from HSTS, and from Redirect Rules or Page Rules that contradict each other. Review those too.

Test it:

curl -sIL -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" http://example.com/

You want a single redirect to the HTTPS address and a 200. See Redirect Chains, Loops, and Incorrect HTTP Status Codes for reading longer chains.

What success looks like: The HTTP address redirects once to HTTPS, and the HTTPS page returns 200 with no loop.

Step 2: Review bot protection

Bot Fight Mode is a free feature that detects traffic matching known bot patterns and issues challenges. Cloudflare's documentation lists limits that matter for SEO work: it cannot be skipped with WAF custom rules or Page Rules, because it does not run on the same rules engine, so "Skip" and "Allow" actions have no effect on it. Its documentation says it may challenge API or mobile app traffic, and it points to Super Bot Fight Mode when you need exceptions for your own clients or monitoring tools.

Its documentation does not say how the mode treats search engine crawlers, so do not assume. Test, and read the events: in Cloudflare's security events, filter for requests from known crawler user agents and look for challenge or block actions.

Do not add a rule that allows anything whose user agent says "Googlebot". That header is easy to fake. Verify real crawlers by reverse DNS or published IP ranges, as explained in Verify Googlebot and Bingbot Requests.

What success looks like: Verified crawler requests are not challenged in the security events, and no important URL returns a challenge to them.

Step 3: Keep Under Attack mode for real attacks only

Under Attack mode shows an interstitial page that runs a check before letting a visitor through. Cloudflare describes it as a last resort for when a zone is actively being attacked, and notes it requires JavaScript to pass, temporarily pauses access to the site and can affect analytics. You can limit it to specific paths or traffic sources through a configuration rule rather than the entire zone.

Left on, every new visitor meets the check, and a client that cannot run JavaScript is unlikely to pass it. Cloudflare's page does not say what happens to specific crawlers, so test with your own tools, and turn the mode off when the attack is over.

What success looks like: The mode is off outside an incident, and any time-limited use was scoped to the affected paths.

Step 4: Treat Rocket Loader and similar optimizations with care

Rocket Loader defers loading of JavaScript until after rendering, to show text, images and fonts sooner. Cloudflare says it keeps scripts in their original order, but also tells you to disable it and retest if you see JavaScript or jQuery problems, and to update your headers if the domain has a Content Security Policy. See Write a Content Security Policy Without Breaking Your Site.

Search engines render JavaScript, so a script error that breaks content can affect what they see. Compare the page with the setting on and off, check the browser console (see Find and Fix JavaScript Console Errors on Page Load), and keep it only if nothing breaks. Cloudflare also notes that Rocket Loader and Email Obfuscation use non-standard tags that fail strict HTML validators without affecting visitors, so do not treat a validator error alone as proof of a problem.

What success looks like: The rendered page, forms, menus and any structured data on your key templates are identical with and without the optimization.

Step 5: Read the managed robots.txt before enabling it

Cloudflare can generate robots.txt directives from your AI bot policies. When your origin already serves a robots.txt, Cloudflare puts the managed rules before your own and serves both as one file; if you have none, it creates one. Its example disallows named AI training crawlers such as GPTBot, ClaudeBot and CCBot, and Cloudflare's page does not present that list as complete. The documentation states compliance is voluntary: the file expresses your preferences but does not technically block access.

Read the merged file at https://example.com/robots.txt afterwards. Confirm your own Disallow rules are still there, that the Sitemap: line is intact and that nothing disallows Googlebot or Bingbot by accident. Robots.txt Blocking Google or Bing? Diagnose It Step by Step walks through the checks.

What success looks like: The served robots.txt contains the rules you intended, and the Googlebot and Bingbot paths you care about are allowed.

Final verification checklist

HTTP redirects once to HTTPS, and HTTPS returns 200 with no ERR_TOO_MANY_REDIRECTS.
The SSL/TLS mode is Full or Full (strict) unless you have a documented reason, and the origin certificate is valid.
Security events show no challenge or block aimed at verified search crawlers on important URLs.
Under Attack mode is off, or scoped and temporary.
Rocket Loader is off or has been tested with your key templates.
The served robots.txt includes your rules and a sitemap line, and does not block Googlebot or Bingbot.
Search Console shows no new fetch or server errors after your change.

Common mistakes to avoid

Leaving SSL on Flexible with an origin redirect to HTTPS. Switch to Full or Full (strict), or remove the origin redirect, to stop the loop.
Allowlisting by user agent. Anyone can claim to be Googlebot. Verify by reverse DNS or published IP ranges.
Expecting a WAF rule to exempt traffic from Bot Fight Mode. It does not run on that engine; use Super Bot Fight Mode if you need skip rules.
Leaving Under Attack mode on. It is a last resort, intended for a zone under active attack.
Turning on several optimizations at once. If something breaks you will not know which one caused it. Change one thing and test.
Assuming managed robots.txt enforces anything. It is advisory; compliance is voluntary.

Frequently asked questions

Does Cloudflare hurt SEO?

Not by itself, and no setting here is a ranking factor. Problems come from specific settings, such as redirect loops or challenges shown to crawlers. Check each one as above.

Is it safe to use Bot Fight Mode with Googlebot?

Cloudflare's Bot Fight Mode page does not say how the mode treats search engine crawlers, so test your own site and read the security events rather than assuming either way. If you need exceptions, Cloudflare points to Super Bot Fight Mode.

What does an Under Attack mode challenge do to a crawler?

Cloudflare's page does not address crawlers. It says the interstitial requires JavaScript, so treat it as a likely barrier for tools that do not run scripts, and use it only when necessary.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.