@charles_irl: my gut says that to solve float numerics problems from nondeterminism x nonassociativity, we need to think bigger than …
Summary
This tweet discusses the idea of training models with 'implementation noise' to improve robustness against float numerics problems caused by nondeterminism and nonassociativity.
View Cached Full Text
Cached at: 05/24/26, 06:24 PM
my gut says that to solve float numerics problems from nondeterminism x nonassociativity, we need to think bigger than determinism.
models could eg be trained with large amounts of “implementation noise” so that the learned network is more robust to implementation skew.
Similar Articles
@charles_irl: Low-precision floats are weird. I have been building up my intuition by playing with them outside of inference/training…
A tweet thread introduces a visualizer for micro-scaling/block quant formats like NVFP4 and MXFP4, explaining how these low-precision floats work and their use in LLM inference to reduce memory bandwidth demands.
A philosophical reflection on the problem of approximation in mathematics, finance, and AI, discussing how models are imperfect but useful, and the implications for AI safety and machine consciousness
A philosophical reflection on the problem of approximation in mathematics, finance, and AI, discussing how models are imperfect but useful, and the implications for AI safety and machine consciousness.
@jchudnov: Pass@k and self-consistency work great for math and code; sample more and verify. So we asked: can the same trick scale…
A new paper shows that scaling inference compute via methods like self-consistency improves LLM accuracy in math and code but fails to improve truthfulness in domains without external verifiers, as model errors are too correlated.
@FrancoisChauba1: If you train on (unsorted list, bubble sort procedure, sorted list) traces, you will never test time compute (TTC) your…
A critique arguing that training LLMs on human-generated data limits their ability to discover novel solutions via test-time compute, and that true AGI requires models that can explore hypothesis spaces more broadly, similar to AlphaZero.
@BetaTomorrow: https://x.com/BetaTomorrow/status/2076323776518906204
This article presents a mathematical perspective on why modern neural networks accommodate diverse architectures and attention mechanisms, framing them as different implementations of constraints within a learnable numerical system.