75% of crawling issues come from two common URL mistakes

75% of crawling issues come from two common URL mistakes

Google discussed its 2025 year-end report on crawling and indexing challenges for Google Search. The biggest issues were faceted navigation and action parameters, which accounted for about 75% of the problems, according to Google’s Gary Illyes. He shared this on the latest Search Off the Record podcast, published this morning.

What is the issue. Crawling issues can slow your site to a crawl, overload your server, and make your website unusable or inaccessible. If a bot gets stuck in an infinite crawling loop, recovery can take time.

  • “Once it discovers a set of URLs, it cannot make a decision about whether that URL space is good or not unless it crawled a large chunk of that URL space,” Illyes said. By then it is too late and your site has slowed to a halt.

The biggest crawling challenges. Based on the report, these are the main issues Google sees:

  • 50% come from faceted navigation. This is common on ecommerce sites, where endless filters for size, color, price, and similar options create near-infinite URL combinations.
  • 25% come from action parameters. These are URL parameters that trigger actions instead of meaningfully changing page content.
  • 10% come from irrelevant parameters. This includes session IDs, UTM tags, and other tracking parameters added to URLs.
  • 5% come from plugins or widgets. Some plugins and widgets generate problematic URLs that confuse crawlers.
  • 2% come from other “weird stuff.” This catch-all category includes issues such as double-encoded URLs and related edge cases.

Why we care. A clean URL structure without bot traps is essential to keep your server healthy, ensure fast page loads, and prevent search engines from getting confused about your canonical URLs.

The episode. Crawling Challenges: What the 2025 Year-End Report Tells Us.


Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.


Barry Schwartz

Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.

In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the “US Search Personality Of The Year,” you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O’Clock.

Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.



#crawling #issues #common #URL #mistakes

Leave a Reply

Your email address will not be published. Required fields are marked *