Robots.txt Generator

Create a robots.txt file that expresses crawler access rules clearly, limits crawling only where intended, and includes a sitemap reference when your site uses one.