Google’s John Mueller responded to a query on LinkedIn to debate using an unsupported noindex directive on the robots.txt of his personal private web site. He defined the professionals and cons of search engine assist for the directive and supplied insights into Google’s inside discussions about supporting it.
John Mueller’s Robots.txt
Mueller’s robots.txt has been a subject of dialog for the previous week due to the final weirdness of the odd and non-standard directives he used inside it.
It was virtually inevitable that Mueller’s robots.txt was scrutinized and went viral within the search advertising group.
Noindex Directive
The whole lot that’s in a robots.txt known as a directive. A directive is a request to an online crawler that it’s obligated to obey (if it obeys robots.txt directives).
There are requirements for tips on how to write a robots.txt directive and something that doesn’t conform to these these requirements is prone to be ignored. A non-standard directive in Mueller’s robots.txt caught the attention of somebody who determined to put up a query about it to John Mueller by way of LinkedIn, to know if Google supported the non-standard directive.
It’s a great query as a result of it’s simple to imagine that if a Googler is utilizing it then perhaps Google helps it.
The non-standard directive was noindex. Noindex is part of the meta robots normal however not the robots.txt normal. Mueller had not only one occasion of the noindex directive, he had 5,506 noindex directives.
The search engine optimization specialist who requested the query, Mahek Giri, wrote:
“In John Mueller’s robots.txt file,
there’s an uncommon command:
“noindex:”
This command isn’t a part of the usual robots.txt format,
So do you assume it’ll have any influence on how search engine indexes his pages?
John Mueller curious to learn about noindex: in robots.txt”
Why Noindex Directive In Robots.txt Is Unsupported By Google
Google’s John Mueller answered that it was unsupported.
Mueller answered:
“That is an unsupported directive, it doesn’t do something.”
Mueller then went on to clarify that Google had at one time thought-about supporting the noindex directive from inside the robots.txt as a result of it could present a manner for publishers to dam Google from each crawling and indexing content material on the similar time.
Proper now it’s attainable to dam crawling in robots.txt or to dam indexing with the meta robots noindex directive. However you’ll be able to’t block indexing with the meta robots directive and block crawling within the robots.txt on the similar time as a result of a block on the crawl will forestall the crawler from “seeing” the meta robots directive.
Mueller defined why Google determined to not transfer forward with the concept of honoring the noindex directive inside the robots.txt.
He wrote:
“There have been many discussions about whether or not it ought to be supported as a part of the robots.txt normal. The thought behind it was that it could be good to dam each crawling and indexing on the similar time. With robots.txt, you’ll be able to block crawling, or you’ll be able to block indexing (with a robots meta tag, if you happen to permit crawling). The thought was that you may have a “noindex” in robots.txt too, and block each.
Sadly, as a result of many individuals copy & paste robots.txt recordsdata with out taking a look at them intimately (few folks look so far as you probably did!), it could be very, very simple for somebody to take away vital components of an internet site by accident. And so, it was determined that this shouldn’t be a supported director, or part of the robots.txt normal… most likely over 10 years in the past at this level.”
Why Was That Noindex In Mueller’s Robots.txt
Mueller made clear that it’s unlikely that Google would assist that tag and that this was confirmed about ten years in the past. The revelation about these inside discussions is attention-grabbing however it’s additionally deepens the sense of weirdness about Mueller’s robots.txt.
See additionally: 8 Common Robots.txt Issues And How To Fix Them
Featured Picture by Shutterstock/Kues
