Google offers each hostname its personal allotted crawl finances. So which means every area, subdomain, and so on has its personal distinctive crawl finances.
This was introduced up within the feedback are in a LinkedIn submit the place Tony McCreath requested a Google crawl finances questions associated to crawling a web page on area A that hundreds a useful resource from area B (completely different server).
John Mueller from Google responded, saying, “They’re unbiased.”
Then former Googler, Pedro Dias wrote, “Afaik, it is allotted set by hostname, proper? So, even throughout the identical area, a number of subdomains may need their very own crawl-rate limits.”
Gary Illyes from Google confirmed this assertion to be true, replying, “right.”
Right here is the submit:
Need to be taught extra about crawl finances, see this Googl help document.
Discussion board dialogue at LinkedIn.
