Robots Exclusion Protocol

The Robots Exclusion Protocol (REP) is a web standard that lets site owners tell automated crawlers which parts of a site should or shouldn’t be accessed by publishing a plain-text robots.txt file at the site root (e.g., /robots.txt). It uses directives like User-agent, Disallow, and Allow (plus nonstandard ones such as Crawl-delay) to set per-crawler rules; compliance is voluntary rather than legally enforceable. REP is separate from page-level controls like the robots meta tag or X-Robots-Tag header, which govern indexing/serving behavior rather than crawl access.

The Robots Exclusion Protocol began in early 1994 when Martijn Koster proposed a simple “robots.txt” convention on the www-talk list to stop ill-behaved crawlers from overloading sites, leading to a community consensus document on June 30, 1994 and rapid adoption by early search engines. Over the years it remained a de facto standard (documented at robotstxt.org) and was interpreted similarly by major engines, with Microsoft, Yahoo, and Google coordinating on consistent behavior by 2008. In July 2019 Google—working with Koster and others—pushed to formalize REP at the IETF, resulting in an Internet-Draft and, ultimately, the official specification as RFC 9309 published on September 12, 2022.

License: IETF Trust Legal Provisions (TLP)

Tags: Robots, Crawling

Properties: User-agent, Disallow, Allow, Sitemap, Crawl-delay, Host, Clean-param, Request-rate, Visit-time, noindex, nofollow.

Website: https://www.rfc-editor.org/rfc/rfc9309.html