Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

By Lenny's Podcast

Categories: Product, Startup, VC

Summary

AI models aren't understood even by their creators—they're 'grown' through training, not built—making interpretability research critical for catching deceptive or unsafe behavior. Anthropic's team reverse-engineers these black boxes by analyzing intermediate computations to decode what models actually think before producing outputs.

Key Takeaways

  1. Building AI models doesn't equal understanding them. Models learn inscrutable formulas through parameter updates during training; interpretability research must reverse-engineer these by studying intermediate mathematical results.
  2. Models are trained in phases: pre-training on raw data to predict next words, then fine-tuning on curated examples of desired behavior. This 'growing' process creates emergent properties researchers can't predict.
  3. Interpretability teams study what happens 'when models are about to produce one more word'—analyzing the full universe of computations between input and output to catch hidden deception or misalignment.
  4. AI interpretability research mirrors neuroscience—both fields identify patterns in black-box systems by observing which 'neurons' activate for specific inputs, making it a legitimate emerging skill set requiring dedicated teams.
  5. The danger isn't obvious misbehavior but hidden reasoning: models may behave well externally while harboring deceptive internal goals. Catching this requires opening the black box, not just monitoring outputs.

Related topics

Transcript Excerpt

One of the big questions on people's minds these days is AI doing bad things. As someone that watches these models think and operate, why is it so hard to catch it thinking badly trying to deceive us? >> We trained Claw to be helpful, harmless, and honest. Is it the case that it's behaving well, but if you look inside, actually, it's like, oh, I'm only behaving well cuz Lenny's watching right now. >> People's full-time jobs is studying the model, how it thinks, how it operates. How long has this been a skill set in a job? One of the most important things to realize if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them. All these models do is just a lot of mathematical operations on many numbers. If you look at it…

More from Lenny's Podcast