Google Updates Its Google Crawlers and Fetchers Documentation

Google has utterly reorganized its crawlers and user-triggered fetchers documentation. It used to be all on one web page and now it’s in a number of pages. A lot of the modifications had been simply shifting content material round however Google did add sections for what product every crawler impacts, and added a robots.txt snippet for every crawler to display easy methods to use the person agent tokens.

Google wrote, “The documentation grew very lengthy which restricted our means to increase the content material about our crawlers and user-triggered fetchers,” so that’s the reason they redid it.

If you happen to dig into every crawl, for instance Googlebot you will note two new sections:

(1) Affected merchandise

(2) Instance robots.txt group

Here’s a pattern screenshot however what I highlighted in purple was added for each crawler:

Right here is how every crawler impacts merchandise:

• Googlebot: Crawling preferences addressed to the Googlebot person agent have an effect on Google Search (together with Uncover and all Google Search options), in addition to different merchandise similar to Google Pictures, Google Video, Google Information, and Uncover.
• Googlebot Picture: Crawling preferences addressed to the Googlebot-Picture person agent have an effect on Google Pictures, Uncover, Google Video, and all options in Google Search the place photographs, logos, and favicons are introduced.
• Googlebot Video: Crawling preferences addressed to the Googlebot-Video person agent have an effect on video-related Google Search options and different merchandise depending on movies.
• Googlebot Information: Crawling preferences addressed to the Googlebot-Information person agent have an effect on all surfaces of Google Information (for instance, the Information tab in Google Search and the Google Information app).
• Google StoreBot: Crawling preferences addressed to the Storebot-Google person agent have an effect on all surfaces of Google Buying (for instance, the Buying tab in Google Search and Google Buying).
• Google-InspectionTool: Crawling preferences addressed to the Storebot-Google person agent have an effect on Search testing instruments such because the Wealthy Outcome Check and URL inspection in Search Console. It has no impact on Google Search or different merchandise.
• GoogleOther: Crawling preferences addressed to the GoogleOther person agent do not have an effect on any particular product. GoogleOther is the generic crawler which may be utilized by varied product groups for fetching publicly accessible content material from websites. For instance, it might be used for one-off crawls for inner analysis and growth. It has no impact on Google Search or different merchandise.
• GoogleOther-Picture: Crawling preferences addressed to the GoogleOther-Picture person agent do not have an effect on any particular product, much like GoogleOther. GoogleOther-Picture is the model of GoogleOther optimized for fetching publicly accessible picture URLs.
• GoogleOther-Video: Crawling preferences addressed to the GoogleOther-Video person agent do not have an effect on any particular product, much like GoogleOther. GoogleOther-Video is the model of GoogleOther optimized for fetching publicly accessible video URLs.
• Google-CloudVertexBot: Crawling preferences addressed to the Google-CloudVertexBot person agent have an effect on crawls requested by the location house owners’ for constructing Vertex AI Brokers. It has no impact on Google Search or different merchandise.
• Google-Prolonged: Google-Prolonged is a standalone product token that net publishers can use to handle whether or not their websites assist enhance Gemini Apps and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise. Google-Prolonged doesn’t impression a web site’s inclusion or rating in Google Search.
• APIs-Google: Crawling preferences addressed to the APIs-Google person agent have an effect on the supply of push notification messages by Google APIs.
• AdsBot Cell Net: Crawling preferences addressed to the AdsBot-Google-Cell person agent have an effect on Google Advertisements’ means to verify net web page advert high quality.
• AdsBot: Crawling preferences addressed to the AdsBot-Google person agent have an effect on Google Advertisements’ means to verify net web page advert high quality.
• AdSense: Crawling preferences addressed to the Mediapartners-Google person agent have an effect on Google AdSense. The AdSense crawler visits taking part websites so as to present them with related advertisements.
• Google-Security: The Google-Security person agent handles abuse-specific crawling, similar to malware discovery for publicly posted hyperlinks on Google properties. As such it is unaffected by crawling preferences.
• Feedfetcher: Feedfetcher is used for crawling RSS or Atom feeds for Google Information and PubSubHubbub.
• Google Writer Middle: Google Writer Middle fetches and processes feeds that publishers explicitly provided to be used in Google Information touchdown pages.
• Google Learn Aloud: Upon person request, Google Learn Aloud fetches and reads out net pages utilizing text-to-speech (TTS).
• Google Web site Verifier: Google Web site Verifier fetches Search Console verification tokens.

Google wrote:

Reorganized the documentation for Google’s crawlers and user-triggered fetchers. We additionally added express notes about what product every crawler impacts, and added a robots.txt snippet for every crawler to display easy methods to use the person agent tokens. There have been no significant modifications to the content material in any other case.

Discussion board dialogue at X.

Picture credit score Lizzi Sassman