Robots.txt Generator & AI Crawler Control
Build valid robots.txt rules, manage AI scraper directives (GPTBot, ClaudeBot, PerplexityBot), and test path permissions live.
Standard Crawler Rules (User-agent: *)
Define the default crawl policies for search engines like Googlebot and Bingbot.
AI Crawler & LLM Training Controls
Toggle specific directives for LLM bots. Block training scrapers while allowing search discovery.
GPTBotOpenAI model training scraper
ClaudeBotAnthropic model training crawler
PerplexityBotPerplexity AI search & answer citations
CCBotOpen-source large web corpus scraper
Google-ExtendedGoogle Gemini training (doesn't affect Google Search)
Live URL Rule Tester
Test whether a specific path is permitted or blocked under your generated rules.
Matched directive: Disallow: /admin/
robots.txt Output
# robots.txt generated via Prospetra Free Web Engineering Suite # https://prospetra.com/tools/robots-txt-generator User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /cart/ Disallow: /checkout/ # OpenAI GPTBot (OpenAI model training scraper) User-agent: GPTBot Disallow: / # Anthropic ClaudeBot (Anthropic model training crawler) User-agent: ClaudeBot Disallow: / # PerplexityBot (Perplexity AI search & answer citations) User-agent: PerplexityBot Allow: / # Common Crawl CCBot (Open-source large web corpus scraper) User-agent: CCBot Disallow: / # Google-Extended (Google Gemini training (doesn't affect Google Search)) User-agent: Google-Extended Allow: / Sitemap: https://example.com/sitemap.xml
Related Free Web Engineering Tools
Complementary developer tools to test, optimize, and validate your web infrastructure.
XML Sitemap Generator & Validator
Validate URLs and generate standards-compliant sitemap.xml files ready for Google Search Console and Bing Webmaster Tools.
Meta Tag & SERP Preview Generator
Craft high-CTR title tags, meta descriptions, and Open Graph tags with live desktop & mobile Google SERP pixel-width previews.
Schema Markup Generator (JSON-LD)
Create rich snippet structured data for FAQPage, LocalBusiness, and Article with real-time JSON-LD schema syntax validation.
Need this fixed directly on your site?
Our software engineers repair code, schema, sitemaps, Core Web Vitals, and accessibility issues directly in your repository. Fixed scope, agreed upfront.