Computers store numbers in fixed-size slots. A signed 64-bit integer tops out at about 9.22×1018. Add past that and the value wraps around, often to a negative number.
In Bitcoin
Amounts are stored in satoshis as 64-bit integers. In August 2010 a transaction in block 74,638 had two outputs of 92,233,720,368.54 BTC each; their sum overflowed, so the old code's "outputs must not exceed inputs" check passed (CVE-2010-5139).
The fix
Today Core checks every output and the running total against MAX_MONEY (21 million BTC) before doing anything else.