Robotstxt Ai

STDIOregistry active

Summary

Gives Claude the ability to analyze, generate, and check robots.txt files with a focus on AI crawler detection. Exposes five tools: fetch_robots pulls and parses robots.txt from any URL, analyze_robots checks pasted content for bot blocking rules, generate_robots creates new files with custom bot selections, list_ai_bots returns a database of 25+ crawlers including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot, and check_bot_status verifies if a specific bot is blocked on a given site. Useful when you need to audit AI crawler access across domains, quickly generate blocking rules without memorizing user agent strings, or help clients understand which bots are hitting their sites. The bot database is the real value here since these user agents change frequently and most devs don't track them.

CodeRabbit

AI writes the code. CodeRabbit catches the slop.

Try For Free →

Keep your Mac awake

Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.

One time payment $9 →

Context.dev

Integrate web data into your AI product. One API to scrape website & brand data.

Get API Key Now →

Make your agent a DeFi expert

Agent, run crypto. Access onchain data & trade routes via 1inch.

Install now →

Make money from your Skills

On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.

Start earning →

AppSignal

Monitor with ease. Code with confidence.

Start Free Trial →

CodeRabbit

AI writes the code. CodeRabbit catches the slop.

Try For Free →

Keep your Mac awake

Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.

One time payment $9 →

Context.dev

Integrate web data into your AI product. One API to scrape website & brand data.

Get API Key Now →

Make your agent a DeFi expert

Agent, run crypto. Access onchain data & trade routes via 1inch.

Install now →

Make money from your Skills

On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.

Start earning →

AppSignal

Monitor with ease. Code with confidence.

Start Free Trial →

npm GitHub stars License

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

Visual Toggle UI — Block or allow AI bots with simple on/off switches
20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
Generate robots.txt — Create a complete robots.txt with your chosen rules
MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

Tool	Description
`fetch_robots`	Fetch and analyze a robots.txt from any URL
`analyze_robots`	Analyze pasted robots.txt content for AI bot blocking status
`generate_robots`	Generate a robots.txt with specified blocked bots and custom rules
`list_ai_bots`	List all known AI bots with user-agents, companies, and descriptions
`check_bot_status`	Check if a specific bot is blocked on a given website

Installation

Web App

git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

MCP Server — Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Tracked Bots

The server knows about 25+ bots including:

AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

Feature	robotstxt.ai	Manual Editing
Visual toggle UI	Yes	No
20+ AI bots database	Yes	Research yourself
Analyze existing robots.txt	Yes	No
MCP Server	Yes	No
Price	Free	Free but tedious

More Open Source SEO Tools

Tool	Description
awesome-seo-mcp-servers	Curated list of SEO MCP servers and agent skills
indexnow-mcp	Instant URL indexing via IndexNow
schema-gen	Schema.org JSON-LD markup generator
ai-visibility	AI brand visibility tracker