Summary
This walks you through the unglamorous work of figuring out how your LLM system actually fails. You review about 100 traces, note what went wrong in each one, then group similar failures into 5-10 categories you can measure and fix. The process is deliberately manual at first because pre-defined failure lists cause confirmation bias. It pushes you to distinguish root causes (missing a filter in the SQL) from symptoms (wrong results downstream) and only build evaluators for failures that warrant the effort. Use it when starting evals, after big pipeline changes, or when production metrics tank and you need to know why.
Install to Claude Code
npx -y skills add hamelsmu/evals-skills --skill error-analysis --agent claude-codeInstalls into .claude/skills of the current project.
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →
Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →
Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →
On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →
Files
SKILL.md
Select a file.
Featured
Keep your Mac awake while Claude Code and 40+ AI agents run. Sleeps when they're idle.
One time payment $9 →Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →On Capafy, your Skill runs online 24/7 as an agent product, and you get paid every time someone uses it.
Start earning →
