Google Says Why It May Ignore Robots.txt And Negatively Impact SEO

Google’s John Mueller answered a query about robots.txt and defined an easy-to-miss mistake that may influence your search engine marketing and web site indexing objectives. The precise challenge was associated to look field spam getting listed by Google, however this error can occur to anybody normally below any context.

Web site Search Field Spam

The one that requested the query on Reddit was affected by a search bar spam assault. What spammers do is search with a question that displays their spammy area of interest, they usually add a hyperlink or a web site identify. What occurs subsequent is that the search bar generates a URL that may be referenced to generate the spammy search end result.

And that’s what was taking place to the one that was asking the query. Their response was so as to add a line within the robots.txt file to forestall Google from indexing the file. However Google was indexing these spammy search-generated URLs anyway.

Google Listed Pages Blocked By Robots.txt

Somebody posted on Reddit that their consumer’s Shopify search field was producing spammy internet pages in response to spammer queries and that Google was indexing them regardless of a robots.txt file prohibiting Google from indexing these pages. What the consumer did was redirect these spammy URLs to a different internet web page. The individual asking the query didn’t ask methods to cease the pages from being listed (which is what they need to have been asking); they requested if these redirected URLs must be marked 404 as an alternative.

The individual asked:

“Engaged on a consumer’s Shopify retailer the place now we have the /search added as a disallow in robots.txt, nevertheless these search outcomes are nonetheless listed within Google.

Nonetheless, if I attempt to open certainly one of these pages, they’ve a redirect set they usually redirect to a different assortment web page on the shop. Ought to we show a 404 web page as an alternative? What’s the best solution to repair this?”

Why Robots.txt File Triggered Spam To Be Listed

Google’s John Mueller took the additional step to establish and evaluate the consumer’s robots.txt file and recognized an error that was inflicting Google to disregard the directive prohibiting Googlebot from indexing search outcomes pages.

Mueller responded:

“Additionally, unsure if it’s your website, however the one I discovered with comparable listed URLs had sections for “user-agent: Googlebot” (within the “START: Customized Guidelines” block in feedback) in addition to much more within the “user-agent: *” part additional down. With robots.txt, the extra particular guidelines win, so in case you have a user-agent: Googlebot part, it would *solely* use that part. If you wish to apply all the principles within the “user-agent: *” part, that you must copy them. Additionally, if that’s your website, then you may simply checklist all of the user-agents that you just wish to have shared guidelines for collectively, eg:

user-agent: googlebot

user-agent: otherbot

user-agent: imgsrc

user-agent: somethingpt

disallow: /fishes

disallow: /orange-cats

… and so on …”

Consumer-Agent Particular Directives Take Priority

What occurred is that the consumer was counting on Google to observe the directives in a line that’s geared toward all consumer brokers, “user-agent: *”, however as a result of there’s one other part of the robots.txt file that’s particular to Googlebot, Google ignored the “user-agent: *” directives and obeyed the one which particularly addressed Googlebot.

Which will sound like a quirk in the way in which robots.txt works, however it truly is sensible as a result of this permits customers to focus on particular crawlers with distinctive guidelines and goal everybody else with a distinct algorithm.

How To Be Secure From Search Field Spam

WordPress and Shopify each have methods to mitigate search field spam.

WordPress and Shopify each have methods to mitigate search field spam.

Shopify Search Field Spam Mitigation

Shopify’s web site has a tutorial on methods to robotically add a noindex directive to all search outcomes. It will successfully forestall indexing of all search outcomes pages. Nonetheless, it’s essential to not block search pages with robots.txt for this to work.

Utilizing a noindex directive is more practical than utilizing robots.txt as a result of robots.txt doesn’t management indexing; it solely controls crawling.

Shopify instructs:

“You’ll be able to cover pages that aren’t included in your robots.txt.liquid file by customizing the

part of your retailer’s theme.liquid structure file. That you must embrace some metatag code to cease the indexing of specific pages.

Out of your Shopify admin, go to On-line Retailer > Themes.

Discover the theme you wish to edit, click on the … button to open the actions menu, after which click on Edit code.

Within the structure folder, click on the theme.liquid file.

To exclude the search template, paste the next code on a clean line within the

part:

{% if template incorporates ‘search’ %}

{% endif %}”

WordPress Search Field Spam Mitigation

It’s fairly simple to mitigate search field spam with WordPress. Customers of the Yoast, Rank Math, and AIOSEO search engine marketing plugins have their search outcomes pages robotically set to noindex by default. Moreover, some web page builders and themes like Divi (and its Further theme) will robotically not generate spammy phrases and URLs in response to searches and as an alternative will inject a number of sentences saying that the search produced no outcomes.

Robots.txt Information

Efficient search engine marketing requires a variety of data. Robots.txt incorporates some quirks that may trigger it to be much less efficient than meant, so it’s helpful to learn up on the official specs to be able to maintain updated.

Featured Picture by Shutterstock/Stockinq