20th anniversary of the search engine blocker file Robots.txt

Today, June 30, is the 20th anniversary of the robots.txt directive file, which was available to webmasters to prevent search engines from crawling their site pages. The robots.txt file was created by Martin Koster in 1994 while working at Nexor after problems crawling his site. All major search engines at the time, including WebCrawler, Lycos and Tavista, quickly adopted this file. Even after 20 years, all major search engines still support and use it.

Brian Yousry posted on his blog about the 20th anniversary and documented the most common errors in robots.txt. If implemented, these errors can be severely detrimental to your rankings and search marketing success. Translated from the source