Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

By ai.engineer

Categories: AI, Tools

Summary

Legora scaled search from hundreds of thousands to 2 billion legal documents by evolving from single Elasticsearch clusters to PostgreSQL with pgvector, discovering that aggressive table partitioning (4,000+ partitions) and regional data isolation solve enterprise compliance requirements—but architectural simplicity matters more than raw performance at scale.

Key Takeaways

  1. Started with single Elasticsearch cluster for all tenants, then multiplied infrastructure 3-4x to meet regional compliance (US, EU, Asia-Pacific data residency requirements). This forced architectural redesign before hitting true scale limits.
  2. Enterprise clients demanded full physical data isolation and customer-managed encryption keys. This forced migration from Elasticsearch to PostgreSQL with pgvector—sacrificing some retrieval performance for unified system architecture and encryption control.
  3. Used aggressive table partitioning strategy (4,000 partitions) with pgvector disk HNSW indexes instead of BM25, trading retrieval performance for compliance and cost management across billions of document chunks.
  4. Infrastructure complexity emerged as projects scaled—active vs. inactive projects (queried constantly vs. never) created uneven load patterns that 'everything just broke and exploded' when scaled, requiring intelligent resource allocation strategies.
  5. Two distinct search use cases (project-scoped search vs. legal research across billions of documents) require different optimization strategies, suggesting founders should architect for use-case-specific scaling rather than one-size-fits-all solutions.

Related topics

Transcript Excerpt

[music] Okay. Hi everyone. Welcome um to this 20 minute talk about connecting AI to loads of legal documents. My name is Jacob. I'm an engineer at Legora. >> Yeah. And I got to step into frame here. Uh I'm Simon. I'm the CEO and uh CEO and co-founder of Turopuffer, a search engine that we work with Legora and others on. >> Neat. Um super quickly um introduction to Legora. We're a collaborative AI platform for legal work. And so that means we have law firms that are clients and we have uh in-house legal teams that are clients. And they use Legora to do reviews of contracts. They use it to go through an absurd amount of contracts and make sure that they all look good. They use them to create new contracts. They do legal research which means um looking over all potential law and they collabor…

More from ai.engineer