Jack Morris: Stuffing Context is not Memory, Updating Weights is

By ai.engineer

Categories: AI, Tools

Summary

Jack Morris discusses the limitations of current language models in handling context, highlighting the challenges of stuffing information into models and proposing alternative approaches like training knowledge directly into model weights.

Key Takeaways

1. Transformer models have quadratic computational complexity, making large context windows exponentially slower and less effective.
2. Most AI models struggle with niche or company-specific information that isn't in their original training data.
3. Current context injection methods become significantly less performant as context size increases beyond 10,000 tokens.

Topics

• AI Model Architecture
• Large Language Model Limitations
• Context Window Performance
• Machine Learning Inference
• AI Knowledge Injection Strategies

Transcript Excerpt

[music] Let's talk about Chad GBT. I think like Chad GBT knows a lot of things. It's actually extremely impressive. I use it all the time. I used it to help prepare for the presentation. I used it to cook last night. Um, you know, very like growing increasingly dependent. And yet, there's a lot that Chad doesn't know. Like, um, it didn't know why my speaker pass wasn't working when I was trying to get into the building and it uh, if you ask it, did the Blue Jays win the World Series? The answer ...