[ad_1]
Reddit has up to date its robots.txt file, stopping Bing and lots of different engines like google from crawling the positioning. “Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their web site,” a Microsoft consultant informed Search Engine Land.
What occurred. On July 1, 2024, Reddit up to date its robots.txt file stopping many engines like google and AI instruments from crawling the positioning. Reddit did not prevent Google from crawling the positioning, regardless of what some folks might have thought earlier this month. However Reddit did stop most different crawlers from crawling the positioning.
Earlier this morning, Mark Williams-Prepare dinner notified me that Reddit outcomes have been dropping out of the Bing Search index. Then several media outlets started masking the information. I wished to substantiate that Bing’s crawlers have been certainly blocked, as a result of Reddit was utilizing IP detection to indicate engines like google one model of its robots.txt file and people one other model of the robots.txt file – as I explained earlier this month.
Bing has thus stopped crawling new content material on Reddit, which is why once you filter Reddit ends in Bing Seek for the final week, you see nothing:
Microsoft confirmed. A Microsoft spokesperson informed Search Engine Land:
“We respect the robots.txt normal. Bing stopped crawling Reddit after they applied their up to date robots.txt file on July 1, which prohibits all crawling of their web site.”
Reddit assertion. Reddit spokesperson Tim Rathschmidt says in an announcement to The Verge:
“This isn’t in any respect associated to our latest partnership with Google. Now we have been in discussions with a number of engines like google. Now we have been unable to succeed in agreements with all of them, since some are unable or unwilling to make enforceable guarantees relating to their use of Reddit content material, together with their use for AI.”
Why we care. With Reddit securing a licensing deal with Google, Reddit is ready to play hardball with different engines like google and AI instruments. So Reddit has blocked most different engines like google from crawling its content material. In the meantime, Google is driving insane traffic to Reddit today, together with testing special treatment in its search outcomes for them.
It makes you marvel if different massive web sites can attempt to go down this route and the place which may depart smaller publishers and content material producers.
In the meantime, don’t count on to see a lot new Reddit content material being surfaced on Bing within the close to future.
[ad_2]
Source link
