Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

By ai.engineer

Categories: AI, Tools

Summary

Almost no AI-generated skills have evaluations before shipping to production—a critical oversight since human-written skills outperform AI-generated ones by significant margins. Skills improve model performance by ~15% on average, but only when properly tested; without evals, you can't distinguish between skill failures and model limitations.

Key Takeaways

  1. Human-written skills dramatically outperform AI-generated skills; 50,000+ skills on SkillBench show almost zero have evals, making it impossible to assess quality reliably.
  2. Skills improve baseline model performance by ~15% across 100+ coding and productivity tasks, but this only holds when skills are properly evaluated and maintained.
  3. Distinguish between capability skills (temporary, teach models new abilities) and preference skills (durable, encode company-specific workflows)—only preference skills need permanent evals to prevent model degradation.
  4. Keep skills.md files under 500 lines of documentation; skills exceeding this threshold often suffer from bloat and reduced model performance due to context dilution.
  5. User-invoked skills are underutilized for workflow automation (pull requests, documentation staging); they avoid context overhead of model-triggered skills while maintaining reliability.

Related topics

Transcript Excerpt

[music] >> Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents. And we are going to talk about why you should not ship skills without eval. And maybe before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code. So yeah, hopefully every every hand goes up, right? And do you use skills with it? Okay. Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today. And like very important is like why checks fail in productions. And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up an…

More from ai.engineer