Google updates crawlers and user-triggered fetchers documentation

Google has made a collection of updates to its crawlers and user-triggered fetchers documentation, largely breaking out the single-page doc into a number of pages and paperwork. However Google additionally expanded what product every crawler impacts with a brand new part subsequent to every crawler and added a robots.txt snippet for every crawler to display tips on how to use the consumer agent tokens by every crawler.

What Google mentioned. Google posted about these modifications saying:

“Reorganized the documentation for Google’s crawlers and user-triggered fetchers. We additionally added express notes about what product every crawler impacts, and added a robots.txt snippet for every crawler to display tips on how to use the consumer agent tokens. There have been no significant modifications to the content material in any other case.”

“The documentation grew very lengthy which restricted our capacity to increase the content material about our crawlers and user-triggered fetchers.”

What’s new. Moreover for simply shifting plenty of content material round, to prepare the content material. Google additionally added the “affected merchandise” sections and likewise the “Instance robots.txt group” sections. Here’s a screenshot of this utilized to the Googlebot crawler, nevertheless it was added to every particular person crawler:

I pulled out every one for you:

• Googlebot: Crawling preferences addressed to the Googlebot consumer agent have an effect on Google Search (together with Uncover and all Google Search options), in addition to different merchandise equivalent to Google Photographs, Google Video, Google Information, and Uncover.
• Googlebot Picture: Crawling preferences addressed to the Googlebot-Picture consumer agent have an effect on Google Photographs, Uncover, Google Video, and all options in Google Search the place pictures, logos, and favicons are offered.
• Googlebot Video: Crawling preferences addressed to the Googlebot-Video consumer agent have an effect on video-related Google Search options and different merchandise depending on movies.
• Googlebot Information: Crawling preferences addressed to the Googlebot-Information consumer agent have an effect on all surfaces of Google Information (for instance, the Information tab in Google Search and the Google Information app).
• Google StoreBot: Crawling preferences addressed to the Storebot-Google consumer agent have an effect on all surfaces of Google Buying (for instance, the Buying tab in Google Search and Google Buying).
• Google-InspectionTool: Crawling preferences addressed to the Storebot-Google consumer agent have an effect on Search testing instruments such because the Wealthy End result Check and URL inspection in Search Console. It has no impact on Google Search or different merchandise.
• GoogleOther: Crawling preferences addressed to the GoogleOther consumer agent don’t have an effect on any particular product. GoogleOther is the generic crawler which may be utilized by numerous product groups for fetching publicly accessible content material from websites. For instance, it could be used for one-off crawls for inner analysis and improvement. It has no impact on Google Search or different merchandise.
• GoogleOther-Picture: Crawling preferences addressed to the GoogleOther-Picture consumer agent don’t have an effect on any particular product, just like GoogleOther. GoogleOther-Picture is the model of GoogleOther optimized for fetching publicly accessible picture URLs.
• GoogleOther-Video: Crawling preferences addressed to the GoogleOther-Video consumer agent don’t have an effect on any particular product, just like GoogleOther. GoogleOther-Video is the model of GoogleOther optimized for fetching publicly accessible video URLs.
• Google-CloudVertexBot: Crawling preferences addressed to the Google-CloudVertexBot consumer agent have an effect on crawls requested by the location house owners’ for constructing Vertex AI Brokers. It has no impact on Google Search or different merchandise.
• Google-Prolonged: Google-Prolonged is a standalone product token that net publishers can use to handle whether or not their websites assist enhance Gemini Apps and Vertex AI generative APIs, together with future generations of fashions that energy these merchandise. Google-Prolonged doesn’t affect a web site’s inclusion or rating in Google Search.
• APIs-Google: Crawling preferences addressed to the APIs-Google consumer agent have an effect on the supply of push notification messages by Google APIs.
• AdsBot Cell Internet: Crawling preferences addressed to the AdsBot-Google-Cell consumer agent have an effect on Google Adverts’ capacity to verify net web page advert high quality.
• AdsBot: Crawling preferences addressed to the AdsBot-Google consumer agent have an effect on Google Adverts’ capacity to verify net web page advert high quality.
• AdSense: Crawling preferences addressed to the Mediapartners-Google consumer agent have an effect on Google AdSense. The AdSense crawler visits collaborating websites in an effort to present them with related advertisements.
• Google-Security: The Google-Security consumer agent handles abuse-specific crawling, equivalent to malware discovery for publicly posted hyperlinks on Google properties. As such it’s unaffected by crawling preferences.
• Feedfetcher: Feedfetcher is used for crawling RSS or Atom feeds for Google Information and PubSubHubbub.
• Google Writer Heart: Google Writer Heart fetches and processes feeds that publishers explicitly provided to be used in Google Information touchdown pages.
• Google Learn Aloud: Upon consumer request, Google Learn Aloud fetches and reads out net pages utilizing text-to-speech (TTS).
• Google Website Verifier: Google Website Verifier fetches Search Console verification tokens.

Why we care. Studying by these affected product sections could provide help to higher perceive how every crawler impacts numerous points of Google. Some don’t affect Google Search in any respect, whereas others are basic to how Google Search works.

Additionally the brand new robots.txt examples could also be very helpful to you and your improvement groups.