What is robots.txt?

Robots.txt is a text file that websites place in their root directory to communicate with web crawlers and search engine bots. It tells these automated visitors which pages or sections of a site they can or cannot access, following the Robots Exclusion Protocol.

Tool Description

The Robots.txt Checker fetches a website's robots.txt directly from our server - no need to copy and paste the file yourself. Enter a URL, and the server requests robots.txt from that site, parses it, and returns the raw content along with the sitemaps it declares, its crawl delay, and its preferred host.

Features

  • Server-Side Fetching: The server requests robots.txt on your behalf, so you don't need the file locally
  • Sitemap Extraction: Lists every Sitemap: entry declared in the file
  • Crawl Delay & Preferred Host: Surfaces Crawl-delay and Host directives when present