Prospetra Free Engineering SuiteAI Crawler Directives

Robots.txt Generator & AI Crawler Control

Build valid robots.txt rules, manage AI scraper directives (GPTBot, ClaudeBot, PerplexityBot), and test path permissions live.

Standard Crawler Rules (User-agent: *)

Define the default crawl policies for search engines like Googlebot and Bingbot.

Disallow: /admin/Disallow: /api/Disallow: /cart/Disallow: /checkout/
Allow: /

AI Crawler & LLM Training Controls

Toggle specific directives for LLM bots. Block training scrapers while allowing search discovery.

OpenAI GPTBotGPTBot

OpenAI model training scraper

Anthropic ClaudeBotClaudeBot

Anthropic model training crawler

PerplexityBotPerplexityBot

Perplexity AI search & answer citations

Common Crawl CCBotCCBot

Open-source large web corpus scraper

Google-ExtendedGoogle-Extended

Google Gemini training (doesn't affect Google Search)

Live URL Rule Tester

Test whether a specific path is permitted or blocked under your generated rules.

ACCESS DISALLOWED

Matched directive: Disallow: /admin/

robots.txt Output

# robots.txt generated via Prospetra Free Web Engineering Suite
# https://prospetra.com/tools/robots-txt-generator

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /cart/
Disallow: /checkout/

# OpenAI GPTBot (OpenAI model training scraper)
User-agent: GPTBot
Disallow: /

# Anthropic ClaudeBot (Anthropic model training crawler)
User-agent: ClaudeBot
Disallow: /

# PerplexityBot (Perplexity AI search & answer citations)
User-agent: PerplexityBot
Allow: /

# Common Crawl CCBot (Open-source large web corpus scraper)
User-agent: CCBot
Disallow: /

# Google-Extended (Google Gemini training (doesn't affect Google Search))
User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Complementary developer tools to test, optimize, and validate your web infrastructure.

View All Developer Tools
Direct Engineering Support

Need this fixed directly on your site?

Our software engineers repair code, schema, sitemaps, Core Web Vitals, and accessibility issues directly in your repository. Fixed scope, agreed upfront.