A fun one for anyone who thinks floating point is just "a bit imprecise":
>>> (1e16 + 1.0) + -1e16
0.0
>>> (1e16 + -1e16) + 1.0
1.0
(Python's built-in sum on floats has used compensated summation since 3.12, so I spell the additions out.) Same three numbers. Different answers. The reason: a 64-bit float has 53 bits of mantissa, so near 1e16 the spacing between representable numbers is 2. Adding 1.0 to 1e16 lands halfway between two representable values and rounds to even, which gives back 1e16. The 1.0 is simply gone before the -1e16 arrives.
Floating-point addition is commutative but not associative. This matters in practice:
- parallel reductions (GPU, threads, SIMD) change the order of additions, so results differ run to run or machine to machine;
- compilers are not allowed to reorder your sums unless you ask for it with something like
-ffast-math.
Mitigations: sort by magnitude, use pairwise summation, Kahan compensated summation, or math.fsum in Python, which tracks partial sums exactly and returns 1.0 for [1e16, 1.0, -1e16] in either order.
No cosmic rays involved here. For once, the arithmetic is guilty all by itself.