The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO

By 20VC

Categories: VC, Startup

Summary

LangChain's Harrison Chase reveals the inference market is far larger than anticipated, but the real business challenge isn't commoditized models—it's building reliable agent systems that handle non-deterministic LLM behavior across cascading model calls. Founders must embrace rapid iteration and 'delete the product' as models improve monthly.

Key Takeaways

  1. LangChain discovered inference demand was massively underestimated; their engineering salary allocation to coding tools dropped 30% while token usage increased, indicating infrastructure shift priorities.
  2. Agent reliability depends on harnessing non-deterministic models through system architecture, not model quality alone. Small input changes produce vastly different outputs across cascading model calls, creating compounding errors.
  3. Product strategy must anticipate obsolescence: features validated 6 months ago become irrelevant as models improve. LangChain switched from JSON DSL parsing to native tool calling when OpenAI released function_call APIs.
  4. Tool calling evolution exemplifies necessary pivots: early workarounds (JSON generation) required rapid replacement with native model features. Recognize paradigm shifts early and commit engineering resources to new approaches immediately.
  5. When building in uncertain markets, 'educated guesses on persistence' beat perfect planning. Accept that some components won't survive; focus engineering on abstractions and observability (monitoring) that remain valuable regardless.

Related topics

Transcript Excerpt

I think we spend 30% of our engineering salary on coding tools. It's actually gone down the last 2 months. While tokens, the number of tokens that we've used has actually gone up. Today we welcome Harrison Chase, the founder of Lang Chain, multi-billion dollar company. It is a leader in the ability to create and monitor your agents. And today we welcomed him to the hot seat to discuss why it is so hard to build agents today. What no one knows that everyone should know about what is coming with agents. And then my favorite, some really kooky predictions which no one else believes will happen, but he knows will definitely happen. >> I really underestimated the demand for inference. Everything just works. This market is so big. There's just so much demand for it. >> Ready to go. Harrison. Dud…

More from 20VC