Google Recommends Using 304 Status Code To Conserve Crawl Budget

Google up to date its crawl price range documentation with new details about how Google’s completely different crawlers all share an internet site’s crawl capability and added a advice to make use of a 304 server response to scale back the quantity of server sources utilized by crawlers.

Each of those additions are to Google’s Optimize Your Crawl Price range Documentation that’s for enterprise web sites with over 1,000,000 pages and medium sized web sites of over 10,000 internet pages that quickly change. Though the documentation shouldn’t be geared toward each web site, the knowledge on that web page is helpful for understanding how crawling works.

Google’s Crawlers Share A Web site’s Crawl Capability

Google added the next passage:

“Whereas every crawler has a special crawl demand, the crawl capability restrict is shared throughout all crawlers. Because of this excessive demand from one crawler can cut back the capability obtainable for others.”

Because of this if a web site will get crawled by Googlebot-Picture and Googlebot, each of the crawlers are sharing the crawl price range for the web site. The impression is {that a} excessive charge of crawling by one bot can cut back the capability obtainable to different Google crawlers.

Google Recommends 304 Server Response

The opposite fascinating advice that Google added was the suggestion to make use of 304 Not Modified server response code. In keeping with the Mozilla Developer Community internet documentation:

“The HTTP 304 Not Modified redirection response standing code signifies that there isn’t a have to retransmit the requested sources.”

The factor concerning the 304 Not Modified response code is that there isn’t a redirection occurring, though technically all 3xx server response codes are classed as redirection codes. Regardless of that technical element, nothing is being redirected.

Google’s up to date documentation says:

“Use HTTP caching: Assist 304 (Not Modified) HTTP standing codes. If a web page hasn’t modified since Google final crawled it, returning a 304 code tells Google to reuse the cached model, saving your server bandwidth and sources.”

A 304 Not Modified response tells Google’s crawlers {that a} web page has not modified because the final time it was crawled. The result’s that the server doesn’t serve the web page to Googlebot and Googlebot can go deal with indexing different pages.

Learn Google’s up to date documentation here. The 2 modifications described listed here are the one new additions. There are different modifications that right here and there that make Google’s documentation extra exact, which is documented here.