Gary Illyes from Google posted on LinkedIn that his mission this yr is to “work out methods to crawl even much less, and have fewer bytes on wire.” He added that Googlebot ought to “be extra clever about caching and inner cache sharing amongst person brokers, and we should always have fewer bytes on wire.”
He added, “Lowering crawling with out sacrificing crawl-quality would profit everybody.”
At this time, Gary added, that Google is crawling as a lot because it did earlier than – regardless of some of us pondering Google is crawling much less. He mentioned “Within the grand scheme of issues that is simply not the case; we’re crawling roughly as a lot as earlier than.”
What Google is healthier at that earlier than is scheduling. “Nonetheless scheduling bought extra clever and we’re focusing extra on URLs that extra prone to deserve crawling,” he defined.
It appears Microsoft Bing, particularly Fabrice Canel from Microsoft and Gary Illyes from Google have the identical objectives. Microsoft is tackling it by encouraging web site homeowners to make use of IndexNow. Google mentioned in November 2021 that Google might consider adopting IndexNow however that got here and went…
John Mueller from Google commented on the publish suggesting, “We might simply crawl RSS feeds and create some type of Reader.” A joke about Google Reader…
Anyway – we’ll see what Google finally ends up doing right here. Right here is his full publish:
My mission this yr is to determine methods to crawl even much less, and have fewer bytes on wire.
A couple of days in the past there was a publish on a Reddit group about how, within the OC’s notion, Google is crawling lower than earlier years. Within the grand scheme of issues that is simply not the case; we’re crawling roughly as a lot as earlier than, nevertheless scheduling bought extra clever and we’re focusing extra on URLs that extra prone to deserve crawling.
Nonetheless, we should always, the truth is, crawl much less. We must always, for instance, be extra clever about caching and inner cache sharing amongst person brokers, and we should always have fewer bytes on wire.
In case you’ve seen an fascinating IETF (or different requirements physique) web draft that would assist with this effort, or an precise customary I would’ve missed, ship it my approach. Lowering crawling with out sacrificing crawl-quality would profit everybody.
Discussion board dialogue at LinkedIn.
