The Problem With Probabilistic Software
Summary
作者指出，过去30年软件建立在确定性假设之上（相同输入产生可预测行为），但AI的概率性系统正在嵌入同样的基础设施，问题在于用治理确定性软件的假设去治理概率性系统，因此需要在AI周围建立新的确定性边界基础设施层。
Similar Articles
We’re putting too much faith in AI’s ability to say no
MIT Technology Review 分析指出，业界过度依赖训练 AI 拒绝有害请求的机制，但这种基于概率的拒绝既不可靠，也难以划定合理边界，可能带来重大安全隐患。
@jakevin7: Let me make a prediction: The next phase of the AI era will become "Infra is all you need". AI-generated code is already very powerful, but it's still far from adequate in terms of usability and stability. Recently, OpenAI's subscription system had a huge bug, and the membership system completely broke down. The system…
The author predicts that the next phase of the AI era will shift from model capabilities to infrastructure capabilities, emphasizing infra abilities such as reproducibility, observability, recoverability, and security isolation, believing that stably carrying AI behavior will be the key to competition.
@snowboat84: To add a supplementary note, regarding the phenomena emerging from AI—scaling laws, emergence, double descent, representation geometry—the papers discussing them are already numerous. But there is a big problem: they are all thinking in the way of computer scientists, not physicists. What is a computer sci…
The author comments that current AI research overuses the thinking style of computer science and lacks a physics-based approach, proposing the need to establish an ideal system like 'Cyber Space' to lay a theoretical foundation.
The mathematics of AI uncertainty
AI systems must incorporate probabilistic reasoning to handle uncertainty for robust and trustworthy decision-making, as discussed by Zoubin Ghahramani regarding the mathematics of uncertainty in intelligent systems.
@freeman1266: Software engineering methodology must shift from the traditional 'state perspective' to a dynamical system perspective. The core view advocates that 'attractor logic takes precedence over governance tools (Harness)', that is, first define the structural invariants that the system should converge to in the long term, rather than merely focusing on local constraints and verification. AI, as a high-frequency and directionless perturbing...
The article proposes that software engineering methodology should shift from a state perspective to a dynamical system perspective, emphasizing that attractor logic takes precedence over governance tools. In the AI era, it is necessary to explicitly model state space, attractors, trajectories, and controls to address architectural drift caused by AI as a high-frequency perturbation source.