Table-driven tests (TDT) are a popular form of unit testing that allow you to rapidly iterate over multiple different inputs to your system in a way that is easily readable and understandable. This post will show my approach to TDT in Python and why all the current implementations of it are insufficient.
robots.txt, or the Robot Exclusion Protocol, is one of the oldest protocols on the web. It's a file, usually stored at the top level of a domain, that provides a list of rules which politely informs web crawlers what they are and are not allowed to do. This simple file is a great insight into the kinds of struggles that web administrators have in maintaining their websites.