Martin Kleppmann: Students using AI did not learn much

By Pragmatic Engineer

Categories: Product, Startup

Summary

A study comparing junior engineers found that those using AI assistants experienced little to no learning, revealing a critical trade-off: while AI accelerates task completion, the struggle inherent in problem-solving is essential for genuine skill development and knowledge retention.

Key Takeaways

1. Educational outcomes prioritize the thought process over artifacts—essays matter not for the final product but for forcing learners to grapple with difficult ideas internally, a benefit AI shortcuts eliminate.
2. Junior engineers using AI showed little to no learning compared to control groups, demonstrating that productivity gains come at a direct cost to skill acquisition and conceptual understanding.
3. Productive struggle is a necessary component of learning—not excessive friction, but enough resistance to force genuine engagement with complex problems and ideas.
4. Organizations must balance AI adoption with intentional learning design that ensures engineers still grapple with difficult concepts, even when AI could solve problems faster.

Topics

• AI and skill development trade-offs
• Learning through struggle framework
• Junior engineer competency assessment
• Educational design vs. productivity gains
• AI adoption in technical training

Transcript Excerpt

In academia, the actual artifacts that the students produce, that's not really the point. We ask them to write essays because we want them to go through a thought process which helps them learn something. And it's that thought process and that learning which is really the desired outcome. >> had a recent study where they looked at junior engineers. One group used AI, the other one did not, and [music] they found, unsurprisingly, that the group who used AI, they had little to no learning. >> Some...