Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel
Summary
70% of skill documentation teaches models information they already know—Intel engineer Nikolay Petrov reveals a deterministic framework using fact-scored item matrices to identify which skills actually add value versus just consuming tokens, transforming skill development from guesswork to measurable impact.
Key Takeaways
- Use a two-arm testing methodology comparing agent performance with and without skills loaded to measure actual impact—Petrov's experiments showed 3/5 tasks solved without skills vs 5/5 with skills, proving measurable ROI.
- Split skills into 25-30 fact-scored items with predefined Q&A pairs, then run a judge panel evaluation to catch performance degradation—identify the rare but critical cases where skills actively hurt model capability.
- Build a 2x2 confusion matrix mapping model knowledge state (knows/doesn't know) against skill impact—only 'model can't answer without skill but can with it' represents genuine value; everything else is bloat.
- Prioritize fixing skills that degrade performance over expanding coverage—popular skills rarely harm (1 out of 400 items in Anthropic skills), but smaller/custom skills have higher degradation rates requiring active remediation.
- Distinguish between knowledge skills (teaching new capabilities) and process/steering skills (defining repeatable agent workflows)—both matter but require different evaluation criteria and optimization approaches.
Related topics
Transcript Excerpt
Hello. My name is Nikolai Petrov and we will be talking about skills and agents knowledge of those skills. We'll start it with the simple prompts where we've been playing with the new stuff, new work being done by the agents, repeating those prompts over and over again, and then we start packing those into the skills, creating repeatable, reproducible, portable type of their functionality or capabilities that we can teach agents to work with. And some of those skills are very important, hundreds of stars on GitHub. There are different types of their skills. Some teach models something new, bringing new knowledge or capabilities to the model, but they're also type of their just steering or process skills where we do not bring any additional information but just getting repeat table flow of …
More from ai.engineer
- Rethinking Claude Code Best Practices — Lydia Hallie, Anthropic
- Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse
- What Your Coding Agent Can't See in the Code - Tom Moor, Linear
- How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp
- How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere