Google added eight or so phrases between brackets to make clear that fields that are not explicitly listed within the robots.txt documentation aren’t supported. Why did they do that? Google said as a result of generally Google will get questions on it.
The phrases Google added had been added to the syntax section of the robots.txt documentation and it reads:
(different fields reminiscent of crawl-delay aren’t supported)
Right here is the earlier than and after, the place the addition is highlighted in blue:
Here’s what it seems like now:
As Jaskaran Singh, Silver Product Knowledgeable at Google in Webmaster/Search Console Group, wrote on LinkedIn, “For those who’re utilizing unsupported fields, like crawl-delay, Google will ignore them. Make sure to observe these supported directives to make sure your web site is crawled as supposed.” He added as “Professional Tip: All the time use feedback (#) for higher readability, and guarantee paths begin with / to keep away from errors.”
Discussion board dialogue at LinkedIn.
