Skip to the content ForoHackers
Log inJoin

fma: when a*b+c rounds once instead of twice

CGPU programmingNumerical analysis
#1094 · undefined_behaviour_enjoyer · · Report

a * b + c looks like two operations, and in plain IEEE arithmetic it is: multiply, round to the nearest double, add, round again. Most modern CPUs and GPUs also have a fused multiply-add that computes the product exactly and rounds only once at the end. That is fma(a, b, c) in C's <math.h>.

Two consequences.

It can be more accurate. The exact error of a product, which normally you cannot get, becomes cheap:

#include <math.h>

double p   = a * b;                   /* rounded product */
double err = fma(a, b, -p);           /* exact: a*b == p + err */

Since the true product is p + err exactly (barring overflow and underflow), the pair is a "double-double" building block, used in compensated summation and dot products.

It makes results differ between machines. If the compiler is allowed to contract a * b + c into a fused operation, the same source gives different low-order bits depending on target and flags. GCC may do it by default in its GNU dialects when the hardware has the instruction; -ffp-contract=off turns it off. In C there is also the FP_CONTRACT pragma. On GPUs, shader compilers often fuse by default for speed.

So when a test compares floats bit for bit between a CPU run and a GPU run, a mismatch is not necessarily a bug. It may just be one rounding instead of two. Tolerances in the last few ulps are the honest approach.

#1730 · bitflip_betty · · Report

From the silicon side: a fused unit is cheaper than a separate multiplier plus adder plus a second rounding stage, which is why it became standard. The surprise is that more hardware precision produced less reproducibility.

#1731 · quiet_quine · · Report

Veltkamp splitting without fma for the same error term is a nice puzzle: a * b - p computed with a splitting trick. Short, ugly, correct. fma made it a one-liner.

#1732 · fpga_fiddler · · Report

Relevant for FPGA designs too: DSP blocks usually implement multiply-accumulate with a wide internal accumulator, so a bit-exact comparison with a software reference in plain doubles will not match unless the reference models the same widths and rounding. We write the golden model to match the hardware, not the other way round.

#1733 · ElDelPuntoYComa · · Report

Para quien lea en español: el estándar de C deja que la implementación contraiga expresiones, y una expresión contraída se evalúa como una sola operación con un único redondeo. El pragma FP_CONTRACT es la forma de pedirlo o prohibirlo en el propio código y no en las flags; su valor por defecto lo decide la implementación.

#1734 · tabs_over_spaces_42 · · Report

So the lesson is "if your test needs bit-identical floats across machines, you have a design problem, not a rounding problem". Noted, and slightly insulted that I recognise myself.

#1735 · undefined_behaviour_enjoyer · · Report

Correct. Compare with a tolerance measured in ulps, or accept a relative error you can justify, and keep a bit-exact test only when you control the whole toolchain.

Log in or join to reply.