Google quietly added a brand new bot to their crawler documentation that crawls on behalf of business purchasers of their Vertex AI product. It seems that the brand new crawler might solely crawl websites managed by the positioning house owners, however the documentation isn’t fully clear on that time.
Vertex AI Brokers
Google-CloudVertexBot, the brand new crawler, ingests web site content material for Vertex AI purchasers, not like different bots listed within the Search Central documentation which are tied to Google Search or promoting.
The official Google Cloud documentation affords the next data:
“In Vertex AI Agent Builder, there are numerous sorts of knowledge shops. An information retailer can comprise just one sort of knowledge.”
It goes on to checklist six sorts of information, considered one of which is public web site information. On crawling the documentation says that there are two sorts of web site crawling with limitations particular to every form.
- Fundamental web site indexing
- Superior web site indexing
Documentation Is Complicated
The documentation explains web site information:
“An information retailer with web site information makes use of information listed from public web sites. You possibly can present a set of domains and arrange search or suggestions over information crawled from the domains. This information contains textual content and pictures tagged with metadata.”
The above description doesn’t say something about verifying domains. The outline of Fundamental web site indexing doesn’t say something about web site proprietor verification both.
However the documentation for Superior web site indexing does say that area verification is required and in addition imposes indexing quotas.
Nevertheless, the documentation for the crawler itself says that the brand new crawler crawls on the “web site house owners’ request” so it could be that it gained’t come crawling public websites.
Now right here’s the complicated half, the Changelog notation for this new crawler signifies that the brand new crawler might come to scrape your web site.
Right here’s what the changelog says:
“The brand new crawler was launched to assist web site house owners establish the brand new crawler visitors.”
New Google Crawler
The brand new crawler is known as Google-CloudVertexBot.
That is the brand new data on it:
“Google-CloudVertexBot crawls websites on the positioning house owners’ request when constructing Vertex AI Brokers.
Person agent tokens
- Google-CloudVertexBot
- Googlebot”
Person agent substring
Google-CloudVertexBot
Unclear Documentation
The documentation appears to point that the brand new crawler doesn’t index public websites however the changelog signifies that it was added in order that web site house owners can establish visitors from the brand new crawler. Do you have to block the brand new crawler with a robots.txt simply in case? It’s not unreasonable to contemplate on condition that the documentation is pretty unclear on whether or not it solely crawls domains which are verified to be underneath the management of the entity initiating the crawl.
Learn Google’s new documentation:
Featured Picture by Shutterstock/ShotPrime Studio
