Decoding Googlebot crawl stats data in Google Search Console

Tucked away within the Settings part of Google Search Console is a report few SEO professionals focus on, however I like to observe.

These studies are often known as Crawl Stats.

Right here, you’ll discover an fascinating set of metrics on Googlebot crawl exercise. These metrics are particularly helpful for web sites with hundreds or hundreds of thousands of pages.

Googlebot ‘Crawl stats’ 

Way back, Google Search Console had easy-to-find metrics on Googlebot crawl exercise. Then, it seemingly disappeared.

In actuality, it was tucked away within the Settings part.

The way to entry the crawl stats studies:

• Click on on Settings on the backside of the left navigation.
• Go to the Crawl stats part.
• Click on Open report.

In regards to the crawl stats information

As Googlebot crawls your web site, Google tracks and studies on numerous facets of Googlebot’s exercise and studies on it in Google crawl stats.

That is the place you’ll discover high-level statistics about Google’s crawling historical past in your web site. 

Google says this information is for superior customers

The Googlebot Crawl Stats information shouldn’t be for the technical SEO rookies.

Google particularly says this information is geared toward “superior customers” with hundreds of pages on their web site, which can be why it’s situated in such an uncommon location, unseen by many within the search engine optimization group.

One purpose Google might understand this as a complicated report is that so many issues can affect these metrics, together with community points and cloud supply providers akin to Akamai.

Who will discover crawl stats most helpful?

I discover the Crawl Stats studies much less of an “superior” set of studies however one thing that’s extra helpful to enterprise SEOs with out crawler monitoring instruments akin to Lumar and Botify. 

When doing search engine optimization on an enterprise web site with hundreds or hundreds of thousands of pages, crawler optimization is a crucial job, and crawler exercise metrics present essential perception for outlining motion gadgets.

Smaller websites probably don’t want to fret an excessive amount of about crawler exercise as a result of there’s most likely sufficient crawl funds allotted to your web site to crawl at an acceptable tempo. 

Alternatively, enterprise websites are likely to have much more pages that should be crawled, found, and/or refreshed than Google crawls their web site every day.

Because of this, they have to optimize for crawler exercise, which is a device to assist information subsequent steps.

What to search for on this information

After years of reviewing this information throughout many websites, I’ve one major rule:

• Don’t spend a number of time right here until you see fluctuations and correlations.

Typically these studies are fascinating however not actionable.

Instance fluctuations that I have a tendency to analyze: 

• HTML requests decreased (or spiked) on the similar time Bytes of JavaScript downloaded elevated (or spiked).
• Common response time elevated (or spiked) on the similar time the variety of HTML requests went down (or sudden fall).
• Complete obtain measurement elevated (or spiked), however the variety of HTML requests didn’t change.
• The p.c of requests for discovery (to find new URLs) will increase and the p.c of requests for refresh goes down; nonetheless, you didn’t launch new URLs on the positioning.

When to take a look at this crawl stats

Crawl stats are good to peruse (and log) a minimum of as soon as a month.

They’re particularly good to observe after main releases, akin to a platform migration or main redesign. It will make it easier to shortly perceive how Google is responding to your newly launched modifications.

Bear in mind: When you’ve got a bot monitoring device akin to Lumar or Botify, then this information isn’t as helpful as you’ll discover within the bot monitoring information offered by these instruments.

Caveats in regards to the crawl stats information

Many issues can affect Google’s crawl stats metrics past a standard dev launch.

This implies the search engine optimization workforce, product supervisor(s) and developer(s) should suppose exterior the field when evaluating the fluctuations.

You will need to contemplate what may have prompted a rise or lower in Googlebot crawl exercise, not solely in your launch but in addition inside the community and tech stack.

Adjustments to one thing akin to Akamai may probably impression this information. 

Log the crawl stats information in a spreadsheet

That is information I wish to archive as a result of Google studies such a small window of time.

A fantastic instance of it is a problem I’m dealing with now with a consumer. What’s reported in GSC proper now seems to be like issues are bettering:

Nevertheless, as a result of I’ve metrics from six months in the past, I can say that these metrics are 40% larger than they have been six months in the past.

Whereas they’re trending down, they’re nonetheless worse than they have been previously. The consumer’s problem is that growth has no concept why that is taking place (sadly, fixing that downside is past the scope of this text).

You might suppose to only seize a screenshot. Nevertheless, it makes it very laborious to check over time.

Discover there isn’t any left axis within the chart. You actually can not inform what the strains replicate. (Notice: Numbers do seem on the left/proper axis if you end up solely viewing two metrics within the chart)

As an alternative, drop this information right into a spreadsheet. Then, you have got precise information that may be charted over time, calculated and used to check with different metrics, akin to visits. 

Having the historic information in a single place is usually helpful when discussing main modifications with growth to indicate how significantly better the metrics have been 4-6+ months in the past. 

Bear in mind, growth likes laborious, particular information, so charts with precise numbers on the left/proper axis (or worse, no numbers on the x-axis in any respect) shall be extra helpful to you than charts with various numbers on the x-axis.

Bear in mind, the studies containers are paginated

Although an important metrics you’ll want are probably seen within the default view, most of the report sections are paginated – and so they’re simple to overlook! 

Which metrics to observe and why

Let’s get into the first metrics to look (in a short time) every month, together with a number of ideas to remove motion gadgets from the information:

Complete crawl requests

• View this report in Google Search Console (situated within the high chart).
• Google definition: “The whole variety of crawl requests issued for URLs in your web site, whether or not profitable or not.”
• If this metric goes up or down, evaluate it with common response time and whole obtain measurement (bytes).
  • An apparent purpose for this metric may go up for those who change a number of code or launch a number of new pages. Nevertheless, that’s certainly not the one trigger.

Complete obtain measurement (byte)

• View this report in Google Search Console (situated within the high chart).
• Google definition: “Complete variety of bytes downloaded out of your web site throughout crawling, for the desired time interval.”
• If this metric goes up or down, evaluate it with common response time
• An apparent trigger for this metric to extend is including a number of code throughout hundreds of pages or launching a number of new pages. Nevertheless, that’s certainly not the one trigger.

Common response time (ms)

• Google Search Console Report (situated within the high chart).
• Google definition: “Common response time for all sources fetched out of your web site in the course of the specified time interval.”
• If this metric goes up or down, evaluate with with whole crawl requests and whole obtain measurement (bytes).

Crawl requests breakdown by response 

• View this report in Google Search Console (situated under the highest chart).
• Google definition: “This desk exhibits the responses that Google obtained when crawling your web site, grouped by response kind, as a proportion of all crawl responses…”
• Widespread responses:
  • OK (200).
  • Moved completely (302).
  • Server error (5xx).
  • Different consumer error (4xx).
  • Not discovered (404).
  • Not modified (304).
  • Web page timeout.
  • Robots.txt not obtainable.
  • Redirect error.

Crawl requests breakdown by file kind

• View this report in Google Search Console. 
• Google definition: “The file kind returned by the request. Share worth for every kind is the share of responses of that kind, not the share of of bytes retrieved of that kind.”
• Widespread responses:
  • JSON.
  • HTML.
  • JavaScript.
  • Picture.
  • PDF.
  • CSS.
  • Syndication.
  • Different XML.
  • Video.
  • Different file kind.
  • Unknown (failed requests).

Crawl requests breakdown by crawl goal

• View this report in Google Search Console.
• Two purposes:
• That is an fascinating metric for displays; nonetheless, it solely has a number of helpful use circumstances. For instance:
  • If the p.c of Googlebot exercise that’s for Discovery immediately will increase, however we’re not including URLs to the positioning, then you have got an motion merchandise to determine what’s being crawled that shouldn’t be crawled. 
  • If the p.c of Googlebot exercise that’s for Refresh decreases considerably, however you didn’t take away pages from the positioning, then you have got an motion merchandise to determine why fewer present pages are being crawled.

Crawl requests breakdown by Googlebot kind

• View this report in Google Search Console.
• Google definition: “The kind of consumer agent used to make the crawl request. Google has quite a few consumer brokers that crawl for various causes and have completely different behaviors.”
• It’s an fascinating metric, however not very helpful. It simply exhibits Google continues to be utilizing their desktop crawler. Truthfully, I often ignore these metrics.

You may click on into every metric for extra information

Typically whenever you current any search engine optimization concern to product managers and builders, they typically need to see instance URLs. You may click on on any of the metrics listed on this report and get instance URLs.

An fascinating metric to take a look at is “different file varieties” as a result of it’s not clear what’s within the “different file varieties” class (typically it’s font information).

The screenshot under exhibits the examples report for “different file kind.” Each file listed is a font file (blurred out for confidentiality causes).

On this report of examples, every row displays one crawl request. This implies if a web page is crawled a number of occasions it may very well be listed greater than as soon as within the “examples.” 

As with all Google Search Console studies, it is a information pattern and never each request from Googlebot.

Do you share these metrics with builders and product managers?

These metrics will usually generate certainly one of two ideas: 

• “There’s nothing to take a look at right here.” 
• “What may have prompted that?” 

In my expertise, the solutions to “what prompted that” are likely to require the enter of product managers and/or builders.

When presenting the information and your questions on potential causes for points, bear in mind to obviously clarify that these metrics aren’t consumer exercise and solely characterize Googlebot’s exercise and expertise on the web site.

I discover product managers and builders typically get a bit confused when discussing this information, particularly if it doesn’t match up with different metrics they’ve seen or information they know in regards to the web site. 

By the way in which, this typically occurs for many Google Search Console information conversations.

If there aren’t any Crawl Stats fluctuations or correlations to be involved about, don’t carry it as much as growth, nor product administration. It simply turns into noise and prevents them from specializing in extra important metrics.

What’s subsequent? 

Take a look at your crawl stats to verify there aren’t any spikes or correlations which might be regarding. 

Then, decide how typically you need to take a look at these and arrange programs that immediate you to verify these and different Google Search Console metrics in a scientific, analytical methodology every month.

Whilst you try your Googlebot Crawl Stats, I’ll write Half 4 on this collection that may discuss tips on how to know which URLs it’s best to concentrate on for technical search engine optimization enhancements and specifically, Core Internet Vitals metrics.

Dig deeper

That is the third article in a collection recapping my SMX Superior presentation on tips on how to flip search engine optimization metrics into motion gadgets. Under are hyperlinks to the primary two articles: