robots.txt as an Insight Into Web Administration Wars
robots.txt, or the Robot Exclusion Protocol, is one of the oldest protocols on the web. It's a file, usually stored at the top level of a domain, that provides a list of rules which politely informs web crawlers what they are and are not allowed to do. This simple file is a great insight into the kinds of struggles that web administrators have in maintaining their websites.