This blog post was a super interesting and well-explained read!
Important Things
FP non-associativity
As a consequence of adding two values with exponent bits + a finite number of mantissa bits, information is lost when the exponents differ.
When adding multiple numbers, which bits get lost can be dependent on the ordering of the additions.
Determinism despite FP non-associativity
Determinism can be achieved by guaranteeing the same ordering of additions. i.e. do not rely on atomic adds and destination ordering (ordering at the memory system which usually does not guarantee a specific ordering), but instead bake in the ordering algorithmically (e.g. tree reductions).
Nondeterminism despite fixed reduction ordering
In chase of performance / maximizing utilization, a kernel will have multiple cases depending on the amount of parallel work available to it. Each case is a different implementation of the algorithm.
Changing how the kernel works based on parallelism implies that the amount of parallelism can influence the ordering of FP additions.
The amount of parallelism (concurrent inference requests) is non-deterministic from the perspective of a single-user, and thus the output of that user’s inference is non-deterministic.