Summary
A proper scraping cascade that tries trafilatura first, falls back to requests with rotated user agents, then escalates to Playwright with stealth mode if the site runs JavaScript or basic anti-bot checks. The code is clean and tracks which method succeeded. The anti-bot landscape section is honest about what playwright-stealth actually handles (navigator.webdriver patches, fingerprint evasion) versus what it doesn't (TLS fingerprinting, Cloudflare Turnstile). The async Playwright variant for Jupyter notebooks is a nice touch since sync Playwright breaks in notebook event loops. This won't beat DataDome or sophisticated bot management, but it covers the 80% case where you just need content extraction with reasonable resilience.
Install to Claude Code
npx -y skills add jamditis/claude-skills-journalism --skill web-scraping --agent claude-codeInstalls into .claude/skills of the current project.
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →
Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →
Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →
On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →
Files
SKILL.md
Select a file.
Featured
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →
