CDN for SEO: Set Up a CDN Without Breaking Crawling

A content delivery network (CDN) is a group of servers spread across many locations that stores copies of your files and serves them from a place near each visitor. For a site with visitors in several regions, or a server that struggles under load, a CDN is one of the most effective performance changes you can make. It can also quietly cause search problems if it is set up carelessly: security rules that block crawlers, cached errors that stay live after you fix them, or pages that change without the change reaching visitors.

This guide is for owners and developers who are about to put a CDN in front of a site, or who already have one and are not sure it is helping. You will decide what to cache, set it up in the right order, protect crawler access, and run checks that prove it works. A CDN does not make your site rank higher. It can lower latency and server load, which makes pages faster to load, and speed is only one part of how a page performs in search.

Before you begin

You need access to your DNS settings, because most CDNs work by changing where your domain points. You need an account with the CDN provider, and access to your web server or hosting panel to look at its cache headers. Know your current performance baseline: save a Lighthouse or PageSpeed Insights result and your current time to first byte (see fixing TTFB).

The main risk is downtime from a DNS or certificate mistake, and a smaller risk is cached copies of the wrong content. Lower the TTL on the DNS records you will change a day ahead, make the change at a quiet time, and write down your current DNS records so you can switch back. Test on a subdomain or a low-traffic site first if you can.

How a CDN works, briefly

web.dev describes a CDN as a distributed network of servers that delivers content from locations closer to users than your origin server. Because the connection from the visitor ends at a nearby edge server, requests have lower latency, and web.dev notes that even uncacheable resources can load faster because the CDN keeps warm connections back to your origin.

Caching is usually "origin pull". The first request for a file is a miss, so the CDN fetches it from your origin and stores the response; later requests are hits. The cache hit ratio is hits divided by total requests. web.dev says a new cache starts at zero and about 90 percent is a common target, but that is a general figure, not a rule for your site.

Step-by-step instructions

Step 1: Decide what goes through the CDN

You have two common setups. In the first, you point your whole domain at the CDN, and it sits in front of every page and file. In the second, you serve only static files, such as images, CSS, JavaScript and fonts, from a separate hostname like static.example.com and leave HTML pages on your own server. web.dev describes both, and says the ideal is to point the site's CNAME at the CDN.

The second setup is simpler and carries fewer search risks, since your pages are unchanged. The first setup gives the largest benefit for TTFB because HTML can be cached too. Whichever you choose, keep one canonical host for your pages (see www vs non-www) and do not leave the same page reachable on both the CDN hostname and your own.

What success looks like: You have a written plan: which hostnames use the CDN and which file types are cached.

Step 2: Connect the CDN and keep HTTPS working end to end

Follow the provider's instructions to add your domain, then change the DNS record they specify. Make sure visitors see a valid HTTPS certificate on every hostname, and that the connection from the CDN to your origin is also encrypted and validated, as the provider's setup guide describes. Providers usually offer several encryption modes, so choose the one that checks your origin's certificate and read the provider's documentation for its exact name. See mixed content if pages load some files over HTTP, and the HSTS guide before enabling strict transport rules.

What success looks like: The site loads over HTTPS on all hostnames with no certificate warning, and HTTP still redirects to HTTPS.

Step 3: Set cache rules by content type

Caching rules decide what is stored and for how long. web.dev gives these principles:

Static assets such as images and versioned libraries should get long lifetimes, for example 6 months to a year, as long as their file names change when their content changes.
Dynamic content can be cached briefly, such as a few seconds, to absorb traffic spikes.
Cache-Control: private marks per-user responses that a shared CDN should not store. Responses marked no-store or private are not cached by CDNs.
At least one of Cache-Control: max-age, s-maxage or Expires must be present for a CDN to know how long to keep a response.
Responses that set cookies typically are not cached, so avoid setting cookies on pages that do not need them.
By default the full URL is the cache key. Ignoring irrelevant query parameters, such as referral IDs, avoids duplicate entries and can raise the hit ratio.

Do not cache pages that are personal to a visitor (accounts, carts, checkout). For header details, read Cache-Control headers.

What success looks like: Static files have long cache lifetimes, HTML has a deliberate rule, and personal pages are never shared between visitors.

Step 4: Turn on compression and modern protocols

web.dev recommends compressing text responses with Brotli or gzip, and says CDNs often handle this for you, including compressing on the fly and recompressing cacheable content at a higher level. It also describes HTTP/2 and HTTP/3 support as useful CDN features, and warns that HTTP/2 is stricter about header formats, so test after enabling it. See Brotli and gzip compression and HTTP/2 and HTTP/3.

What success looks like: The response headers show Content-Encoding: br or gzip for text files, and the protocol column in DevTools shows h2 or h3.

Step 5: Keep search engine crawlers able to reach the site

This is the step most CDN guides skip. Security features that challenge or block unfamiliar visitors can also block crawlers. Google's documentation gives three facts to build on.

First, anyone can claim to be Googlebot in the user-agent header, so Google recommends verifying requests with a reverse DNS lookup or by matching the source IP against its published ranges before you trust or allowlist them. The hostname must be under googlebot.com, google.com or googleusercontent.com, and a forward lookup must return the original IP. Google publishes the IP ranges as JSON files for large-scale checks. See verifying Googlebot and Bingbot.

Second, status codes matter. Google treats a 429 response like a server error, and 500, 502 and 503 responses lead it to slow its crawling. Other 4xx responses are treated as the content not existing, and already-indexed URLs can drop out of results over time. Google warns against using 401 or 403 to limit crawl rate, because they do not reduce it. If your CDN's bot protection returns 403 or a challenge page for crawlers, that is the kind of response that can remove pages from search.

Third, a CDN's firewall can block a crawler without any change to your own server, so your server logs may show no sign of it. Review your CDN's security events for blocked requests from the published Google IP ranges, and check Bing's own documentation for how to verify its crawler. Our guide to Cloudflare bot blocking covers one popular provider in detail.

What success looks like: Verified Googlebot requests receive 200 responses for pages and robots.txt, and your rules do not use blanket challenges for search engine crawlers.

Step 6: Test the live result as a search engine would

Use the URL Inspection tool's live test in Search Console on your home page and a deep page. A successful test means Google can reach the page, and the tested page view shows the HTML, headers and screenshot it received. If screenshots differ from what you see, some resources may be blocked. Also fetch robots.txt and your sitemap and confirm they return 200 with the right content type, not a CDN error page or HTML wrapper. See robots.txt blocking search engines.

What success looks like: The live test succeeds, and robots.txt and the sitemap load normally through the CDN.

Step 7: Check cache behavior and purge

Request the same URL twice with curl -I. Most CDNs add a header showing whether the response was a hit or miss; the header name varies by provider, so check its documentation. The second request should be a hit for cacheable files, with an Age header if the CDN sends one.

Practice purging. Edit a page, purge it using the provider's tool, and confirm that the new version appears. web.dev notes that you can purge by URL or, with cache tags, in groups. If you update CSS or JavaScript, change the file name (version it) so you do not depend on purging at all.

What success looks like: A repeat request is served from the cache, and a purge makes a change visible within the time the provider states.

Step 8: Compare against your baseline

Retest PageSpeed Insights and TTFB from the same location as before. Check mobile and desktop. CDN performance varies by geography and time of day, according to web.dev, so test from more than one place and do not rely on a vendor's marketing claims. Field data takes weeks to show real changes.

What success looks like: Your lab numbers are at least as good as the baseline, and nothing in Search Console crawl stats or page indexing has newly worsened.

Final verification checklist

The site loads over HTTPS on every hostname, with a valid certificate on the CDN and origin.
One canonical host is used for pages, and the others redirect to it.
Static files have long cache lifetimes and versioned file names.
Personal pages are marked private or no-store, or excluded from caching.
Text files are compressed, and Vary: Accept-Encoding is present.
Verified crawler requests get 200 responses, not 403, 429 or challenge pages.
robots.txt and the sitemap load normally through the CDN.
The URL Inspection live test succeeds on a deep page.
A purge makes a content change visible.

Common mistakes to avoid

Caching error pages. If the origin has an outage and the CDN stores an error response, it may keep serving it. Check how your provider treats error responses, and purge after an incident.

Challenging all bots. Turning on strict bot protection without allowing verified search crawlers can block indexing. Review it against Google's verification method.

Allowlisting by user-agent only. Anyone can fake it. For Google, verify by reverse DNS or the published IP ranges.

Caching personalized pages. Without private, a shared cache can serve one visitor's page to another.

Long cache times on unversioned files. Visitors may keep old CSS or JavaScript. Version file names or purge.

Leaving two hostnames serving the same pages. This creates duplicates. Redirect extras to the canonical host.

Setting cookies on every response. Cookies can stop pages from being cached, so the CDN gives less benefit.

Frequently asked questions

Does a CDN improve SEO?

A CDN can make pages load faster and reduce server load, and speed contributes to page experience. It does not assure better rankings, and it can cause problems if it blocks crawlers.

Does a CDN change my page URLs?

No, if the CDN sits in front of your own domain, page URLs stay the same. If you serve static files from a separate hostname such as static.example.com, only the file URLs change, not the page URLs.

Should I cache HTML pages?

You can, for pages that are the same for everyone, using short lifetimes or purge-on-change. Do not cache pages that depend on who is logged in.

Do I need a CDN for a small local site?

Not necessarily. If your visitors are near your server and your pages are fast, the benefit may be small. Fix large images, slow hosting and render-blocking files first.

What is Cache-Control s-maxage?

It sets the freshness lifetime for shared caches such as CDNs only, and overrides max-age for them. Browsers ignore it.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.