Google introduced a brand new robots.txt report inside Google Search Console and on the similar time stated it can sundown the outdated robots.txt tester instrument. The brand new instrument reveals which robots.txt information Google discovered for the highest 20 hosts in your website, the final time they had been crawled, and any warnings or errors encountered.
Google additionally stated on X, “We’re additionally making related information obtainable from the Web page indexing report. As a part of this replace, we’re sunsetting the robots.txt tester.”
To entry this, go to the settings hyperlink to your Search Console profile or click here. That is what it appears to be like like:
Here’s what the report appears to be like like for this website, sure, I’ve a essential error. Listed below are these particulars and notice, you possibly can request recrawl of your robots.txt file:
This help document goes by way of much more about this report.
Some aren’t pleased that the robots.txt tester tool is gone however Bing still has one:
Good alternate options to Google’s depreciated instrument to validate your robots.txt:
• Merkle: https://t.co/33yMW9IlPB
• Ryte: https://t.co/jYAGuHN2q5
• Screaming Frog (in-crawl): https://t.co/zsr2KMiNmC— Cyrus website positioning (@CyrusShepard) November 15, 2023
I like the brand new robots.txt reporting (it is in settings btw), however eradicating the robots.txt Tester? I like that instrument and it may be super-helpful. There are different instruments that do comparable issues, together with one from Bing, however I might like to see that keep in GSC. 🙂 https://t.co/hMIJQjfpQV
— Glenn Gabe (@glenngabe) November 15, 2023
Discussion board dialogue at X.
