Summary
This is a practical implementation for crawling documentation sites and blogs while respecting robots.txt and rate limits. It combines trafilatura for clean content extraction with BeautifulSoup for structure parsing, then converts everything to markdown for RAG ingestion. The code handles common doc frameworks like Docusaurus and Sphinx, extracts sidebar navigation and prev/next links, and includes sitemap discovery. Worth noting that it tracks visited URLs and content hashes for incremental updates, which matters when you're maintaining a knowledge base that needs to stay current. The robots.txt checker is solid, and the configurable rate limiting (defaults to 1 second between requests) keeps you from being a bad citizen.
Install to Claude Code
npx -y skills add mindmorass/reflex --skill site-crawler --agent claude-codeInstalls into .claude/skills of the current project.
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →
Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →
Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →
On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →
Files
SKILL.md
Select a file.
Featured
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →First SeenJun 3, 2026
