What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma

By ai.engineer

Categories: AI, Tools

Summary

A $900M bank error reveals why AI agent autonomy is dangerously similar to human decision-making failures: Citibank's Flexcube system had proper controls, but human operators overrode warnings because the transaction *looked* correct in context. As we deploy autonomous agents, we face an identical problem—systems that follow logic but miss intent.

Key Takeaways

  1. Control systems fail not because they're missing, but because operators override them when context appears normal. Citibank's warning fired correctly, but approvers ignored it because interest payments were legitimately supposed to leave the bank—they couldn't distinguish legitimate from erroneous transactions in real-time.
  2. The wash account design was correct in theory but failed in practice: the checkbox wasn't marked, creating a gap between intended logic and execution. For AI agents, this mirrors the challenge of translating high-level intent into unambiguous system behavior across edge cases.
  3. Regulatory controls exist because of failure modes, not theory. After Revlon, loan agreements added clauses preventing lenders from keeping erroneous payments—this retroactive fix teaches us that AI agent guardrails must be built anticipating the specific ways systems will fail, not just theoretically prevent them.
  4. Autonomous agents face the same non-deterministic challenge as humans: you cannot know in advance what they'll do. The real problem isn't lack of controls—it's that giving agents autonomy over complex systems like Flexcube means preventing accidents without over-constraining the system.
  5. Context-dependent decision-making is the vulnerability. Three approvers made the rational decision given available information. As AI agents make decisions across financial, operational, and safety systems, the question becomes: how do we prevent agents from making locally-correct decisions that are globally catastrophic?

Related topics

Transcript Excerpt

[music] >> Uh my name's John Feminella. I work on AI architecture and engineering at Two Sigma, which is a hedge fund here in New York. Before I get too far into things, I have a legal disclaimer slide that I have to show you. It says two things. One, I promise not to sell or market or offer you any securities at any point in this talk. And two, everything in this is my own opinion and not the opinion of Two Sigma. All right, thanks for indulging me. If I don't say those magic words, the 5G microchips implanted in me will explode. So, that slide is a good example, actually, of how our industry is regulated. And as a regulated company with lots of controls, I think one thing that's always interesting is to explore what happens when controls fail, because that gives you a good idea about why…

More from ai.engineer