Crawvix

Faceted Navigation SEO: Stop Filters Creating Endless URLs

Crawvix Editorial Team · Published October 29, 2026 · Last updated October 29, 2026 · 10 min read

Filters are good for shoppers. A visitor can narrow a list of shoes to size 9, black, under a set price, and see only what matters. The trouble starts when each combination of filters gets its own URL. A list with a handful of filters can produce thousands or millions of addresses that all show slightly different slices of the same products. Crawlers cannot tell whether a new filter URL is useful without fetching it, so they may spend their time on combinations nobody searches for.

This guide is for owners and developers of shops, directories, listing sites and any site with filters, sorting or parameters. By the end you will know which filtered pages are worth keeping crawlable, how to keep the others out of the crawl, how to write the robots.txt rules, and how to make the URLs you do keep clean and predictable. It does not promise more indexed pages or better rankings. It aims to stop wasted crawling and duplicate pages.

Before you begin

You need an inventory of how your site creates filter and sort URLs. Look at a category page, apply two filters, change the sort order, go to page two, and write down every URL that results. You also need access to your robots.txt file, your page templates (for canonical and link attributes) and Search Console. If your site is a single-page app, include a developer, since some of the fixes happen in application code.

Plan a few hours to map the URLs and a few weeks for crawlers to respond. The main risk is blocking filter pages that bring in search visits today. Before changing any rule, look in Search Console's Performance report to see which filtered URLs get clicks. Save your current robots.txt first.

Why faceted navigation causes problems

Google's guidance on faceted navigation describes two issues. The first is overcrawling: crawlers request a large number of useless URLs before they recognise them as such. The second is slower discovery: time spent on useless URLs leaves less time for new, useful pages. Google gives an example of a URL like items.shtm?products=fish&color=radioactive_green&size=tiny, where changing any parameter changes the results, so the number of possible URLs grows quickly with each added filter. Google also notes that crawling lots of faceted URLs uses real server and rendering resources.

Parameters that do not change content cause a related problem. Google's URL structure advice lists referral, sorting and session parameters as ways to produce duplicate URLs, and calendars that link forward forever as another source of near-endless URLs. For whether this matters at your scale, see crawl budget explained. Not every site needs to worry, but if Search Console shows many URLs in "Discovered - currently not indexed", it is a reason to look. See Discovered - Currently Not Indexed.

Step-by-step instructions

Step 1: Sort your URLs into three groups

Take your list of URLs and put each pattern in one of three groups.

  • Pages that deserve to be found in search. A category plus a single stable attribute people search for, such as "women's running shoes". These need their own indexable, crawlable URL.
  • Pages that help visitors but not search. Sorted lists, multi-filter combinations, price sliders, "in stock only" toggles.
  • Pages that should not exist. Empty combinations, duplicate filters, nonsense combinations and pages beyond the last page of results.

Look at demand to make the first split. If people do not search for "black running shoes size 9 under 50", that URL has no search value, even though it is valuable to the shopper using the filter.

What success looks like: A table of URL patterns, each marked "index", "crawl-skip" or "return 404".

Step 2: Keep the useful pages crawlable and clean

For the first group, Google's advice for crawlable and indexable facets is:

  1. Use & as the parameter separator. Commas, semicolons and square brackets are often not recognised as separators by crawlers.
  2. Keep path-based filters in a fixed order and block duplicate filters. A URL like /products/fish/green/tiny should not also exist as /products/tiny/green/fish.
  3. Return a 404 for empty or invalid combinations.

Google's general URL guidance adds that you should use = for key-value pairs and & between parameters, remove any parameter that does not change content, avoid session IDs in URLs (use cookies), and convert URLs to one case, since URLs are case-sensitive. Use readable words and hyphens in path segments. See SEO-friendly URLs.

Give these pages unique titles, headings and introductory text so each is more than a filtered list.

What success looks like: The indexable filter pages have one canonical address each, in one parameter order.

Step 3: Keep crawlers out of the low-value combinations

For the second group, Google's first option is to prevent crawling with robots.txt, which it recommends when faceted URLs do not need to appear in Google Search. The idea is to keep crawlers on item pages and a single unfiltered listing page. Google's own example blocks the filter parameters and allows one specific exception. Adapt it to your parameter names:

User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=
Allow: /*?category=all$

How this works:

  • * matches any run of characters, and $ marks the end of the URL.
  • Google applies the most specific (longest) matching rule, and if an Allow and a Disallow are equally specific, the less restrictive one wins.
  • A group for a named crawler is used instead of the * group for that crawler, not in addition to it. If you write a Googlebot group, repeat any rules from your general group that still apply to it.
  • Rules are case-sensitive.

Test sample URLs before you publish. The Search Console robots.txt report confirms that the live file is fetched and parsed, and Google's open-source robots.txt library lets you try URLs against a draft. The robots.txt troubleshooting guide shows how to read results. Keep in mind that a URL blocked in robots.txt can still be indexed without its content if other pages link to it. If a filtered page must stay out of results, use noindex on a crawlable page instead, as explained in noindex vs robots.txt. Google's pagination guidance also mentions noindex or robots.txt as ways to avoid indexing variations of the same list.

What success looks like: Sample filter URLs are disallowed, the exceptions are allowed, and nothing important is blocked.

Step 4: Use weaker signals only as a supplement

Google describes two weaker options, which it says are generally less effective in the long term than blocking.

  • rel="canonical". Pointing filtered URLs at a main version, for example a URL with three filters pointing to the version with one, may gradually reduce crawling of the non-canonical versions. Remember that a canonical tag is a hint for consolidating duplicates, not a crawl control. See duplicate URLs and canonical tags.
  • rel="nofollow" on links to filtered pages. It may help, but every link to that URL must carry the attribute for it to work, including links from elsewhere on your site.

Use these when robots.txt would be too blunt. Do not treat them as a switch that stops crawling.

What success looks like: Where canonical or nofollow is used, it is applied consistently by the template, not page by page.

Step 5: Return 404 for nonsense URLs

For the third group, Google says to return an HTTP 404 for filter combinations with no results, URLs with duplicate filters or nonsensical combinations, and nonexistent pagination URLs. The 404 should come from the URL that was requested. Do not redirect these to a generic error page. Single-page apps may not be able to set the status; the single-page-app guidance in soft 404 errors covers their options.

Pagination needs the same care. Each page in a series should have its own address, such as ?page=2, not a URL fragment, and each should have its own canonical; Google says the first page should not be the canonical for the whole set. See pagination SEO best practices.

What success looks like: A made-up filter combination returns a 404 status, not a 200 with an empty list.

Step 6: Do not rely on URL fragments, but know what they do

Google states that fragments (the part after #) are generally not supported in crawling and indexing, so filters placed in a fragment have no effect on crawling, good or bad. This makes fragments a legitimate way to offer filters that you do not want crawled at all. It also means you must not place filters you do want indexed in a fragment. Google's URL guidance adds that you should not use fragments to change page content, and should use the History API for JavaScript-driven changes.

What success looks like: You know which filters live in fragments or in client-side state, and none of them are meant to be indexed.

Step 7: Check the effect over time

After the changes are live, give crawlers time. In Search Console, open Settings, then Crawl stats (available for Domain and root-level properties) and compare the share of requests by response type and file type before and after. Google says some requests may not be counted, so use trends, not exact figures. Check the Page Indexing report for new "Blocked by robots.txt" entries and confirm they are the patterns you meant to block; see Blocked by robots.txt in Search Console.

What success looks like: Requests to filter and parameter URLs fall over the following weeks, and important category pages are still indexed.

Final verification checklist

  • Every filter and parameter pattern is marked index, crawl-skip or 404.
  • Indexable filter pages use & separators and a fixed parameter order.
  • Parameters that do not change content have been removed from links.
  • Session IDs are not in URLs.
  • robots.txt rules block the intended patterns and allow the exceptions.
  • Sample blocked URLs and sample allowed URLs were tested before publishing.
  • Empty or nonsense filter combinations return a 404 status.
  • Each paginated page has its own canonical URL.

Common mistakes to avoid

  • Blocking filters that earn search visits. Check Performance data first.
  • Using noindex to stop crawling. The page is still fetched. Use robots.txt for crawl control.
  • Relying only on canonical tags. Google says they are slower and less reliable than blocking.
  • Putting nofollow on some links but not others. Every link to the URL must carry it.
  • Using commas or brackets as parameter separators for filters you want indexed. Use &.
  • Letting the same filters appear in any order. Fix the order and reject duplicates.
  • Redirecting empty combinations to the home page or a generic error. Return a 404 at the requested URL.
  • Putting wanted filters in a URL fragment. Fragments are not crawled or indexed.

Frequently asked questions

Should I block filter URLs in robots.txt or use canonical tags?

If the filtered pages do not need to appear in search, Google's first option is robots.txt. Canonical and nofollow are weaker, slower signals. Use canonical when you need to consolidate duplicate-looking pages that must stay crawlable.

Do I need to worry about this on a small site?

Usually only if filters or parameters generate many more URLs than you have real pages. Google says sites without large numbers of rapidly changing pages generally do not need to manage crawl budget. Check Crawl Stats first.

Can I make some filter pages rank?

Yes, if there is real search demand for the combination. Give the page its own clean URL, unique content and internal links, as in Step 2.

What happens if I block a URL that is already indexed?

It can remain in the index without its content if other pages link to it. If you want it removed, make it crawlable with noindex until it drops out. See the noindex guide.

Can I put filters in a URL fragment so they are never crawled?

Google says it generally does not support fragments in crawling and indexing, so filters placed there have no effect on crawling, good or bad. That works for filters you never want indexed, but not for filters you do.

How do I know if my site has too many filter URLs?

Crawl Stats shows how many requests go to which response types and file types, and the Page Indexing report shows how many URLs are known but not indexed. Compare the numbers with the real size of your catalogue.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.

In this category

All 19 Crawlability guides