A short history of robots.txt
In 1994, Martijn Koster proposed a simple convention: before crawling a site, fetch a file called robots.txt and honor what it says. There was no standards body behind it and no enforcement mechanism — just an agreement among the handful of people running crawlers that politeness was cheaper than an arms race.
The convention held for nearly three decades on goodwill alone, which makes it one of the most successful informal protocols on the internet. It was finally formalized as RFC 9309 in 2022, long after every major search engine had treated it as law.
The catch has always been the same: robots.txt is a request, not a gate. A crawler that ignores it suffers no technical consequence. Everything interesting about modern bot policy starts from that asymmetry.