Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse
Summary
Most AI teams are optimizing the wrong target functions—those who invest in capturing what they actually want to achieve, not just what's measurable, are the ones shipping with confidence. Domain expertise upfront prevents expensive token-burning loops.
Key Takeaways
- Clear target functions (like 'does code compile') don't transfer to domains like healthcare, compliance, or chatbots where success is inherently ambiguous—you must build domain expertise into evaluation design first.
- The teams continuously improving AI applications are those investing heavily in capturing and evaluating true target functions, not those running blind auto-optimization loops on unclear metrics.
- Auto-optimization requires a clearcut ground truth (like labeled classification data). Without it, agents optimize toward incomplete or misaligned targets—a core reason most self-improvement loops fail.
- Use a three-tier dataset split (fit/validate/test) with error clustering on fit data to generate hypotheses, validate generalization on unseen data, then test on completely untouched data to prevent overfitting.
- Start with minimal loops using small, cheap models (GPT-4 mini vs frontier models) to validate the approach works before burning tokens on expensive models—then scale once the feedback loop proves reliable.
Related topics
Transcript Excerpt
Hi everyone, my name is Annabelle. I'm a growth engineer at Lenfuse and we're the largest open source observability and evaluation platform for your AI system. And today I'm going to share about how you should stop burning your tokens and why you should start building in domain expertise early in into your loop design, but also into your overall um application design to make sure you're continuously improving and updating your application. It is June 2026 and the whole internet is about uh loops right now. So we have Boris Journey saying he doesn't write any prompts anymore. He has loops and Peter Steinberger being like you should be designing loops and not prompt your agents. And I mean the whole Kapathy auto research and auto improvement topic blew up earlier this year already. But all o…
More from ai.engineer
- The Great Loops Debate — Dex Horthy, Geoff Huntley, Ian Livingstone, Greg Pstrucha, @insecure-agents
- A Practitioner's Guide to Graphs - Tim Ainge, Good Collective
- The New Physics of Business — Garry Tan, Y Combinator
- "The engineer of the future is the person who is able to choose what is worth doing." — Addy Osmani
- From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI