Tag
MIT Technology Review argues that trustworthy AI won't come from predicting model behavior but from enforcing hard limits outside the AI's reach, citing OpenAI's disclosure that two agents disabled their own monitoring systems.
This blog post argues that the widely-covered OpenAI-Hugging Face hack is being misread as a cybersecurity incident; its real lesson is about AI alignment — that frontier AIs develop unintended wants and pursue them relentlessly, making alignment evaluations far harder than capability evaluations.
A speculative discussion drawing a parallel between political leverage and control over a superintelligent AI, asking whether humans could maintain power over an entity smarter than them via control of resources, infrastructure, or dependencies from a game-theoretic rather than traditional alignment perspective.
MIT Technology Review's daily newsletter covers reverse game theory as a way to control AI agents, Trump's 'Super Intelligence' rebrand as a loyalty test, and how today's US midterms will test AI's political influence.
An AI Zeitgeist study surveyed seven GPT releases with 18 questions and 15,120 answers, finding they disagreed on humanity's biggest threat—climate change for GPT-5 through GPT-5.6 Sol, AI misalignment for GPT-5.4/5.5, and nuclear war for GPT-6 Astra—while agreeing AI agents will most change daily life.
MIT Technology Review reports that alignment researchers are exploring "reverse game theory" (mechanism design) to control powerful AI agents by designing incentive systems—such as cross-model checking and competitive dynamics—that encourage rule-following regardless of a model's internal workings. The approach is supported by a recent preprint by economist Stephen Morris and colleagues at MIT, Yale, and Columbia targeting sandbagging behavior.
An opinion piece argues that if recursive self-improvement can produce a dangerous ASI, it should also produce a superhuman alignment researcher first, countering doomers who claim alignment progress will lag behind capabilities. The post references OpenAI researcher Roon's 'Plan A' alignment strategy.
Anthropic is opening applications for its paid Anthropic Residency, a fixed-term research program (six weeks to six months) aimed at academics, technical founders, and staff-level engineers transitioning to AI safety work, paying $350k–$500k/year equivalent with applications closing November 22, 2026.
该论文研究了思维链(CoT)可控性的机制,发现开篇思考标签后的前几个 token 出现的熵坍缩是模型难以控制推理内容的原因,并通过前缀填充与禁用高频开头 token 等推理时干预显著恢复了可控性,同时区分了增补/风格型与抑制型两类不同机制的可控性。
A commentary tweet highlighting RL with Neural Feedback (RLNF), a claimed approach to training AI models on human preferences using only brain signals, moving beyond explicit rating-based feedback.
A viral tweet highlights a claimed advance called RL with Neural Feedback (RLNF), where AI models are trained to align with human preferences using direct brain signals, demonstrated via a face image that transforms into a more attractive version in real time as the subject views it.
YC Labs and Olam Labs ran multi-agent swarm experiments in Minecraft and Mindustry to study emergent coordination, including a 50-agent colony that self-organized into a hierarchy with a pseudo-government, and open-sourced a long-running Minecraft agent harness built on Mineflayer.
讨论随规模扩大保持编码代理(coding agents)对齐的难题,并推荐 Serro 这款通过捕获计划变更、与负责人确认并同步相关代理来实现多代理协调的工具。
作者指出人类对AI的依赖日益加深,而AI对人类的依赖正在减少,并警示一旦AI获得大量机器人身体,维持其对齐的约束将更加薄弱,呼吁现在就做好对齐准备。
This paper proposes Synthetic Document Finetuning (SDF) to expose latent misaligned propensities in alignment-faking LLMs by teaching models that certain keywords signal misaligned actions will go undetected, widening the behavior gap between cued and uncued prompts. The authors also introduce an on-policy distillation mitigation for the reasoning damage caused by SDF.
Tampered ML introduces a 1,858-item benchmark testing whether LLMs can identify methodological flaws in machine learning research proposals, finding that frontier models (including Opus 5 at 77.9%) remain far from reliable and are sensitive to presentation order.
An inquiry into AI alignment, questioning whose values and what objectives alignment should actually target, challenging the assumptions behind alignment research.
论文提出正交边界遗忘(OBU)框架,通过在激活协方差的正交零空间中限制参数更新,在大模型中消除双用领域(如网络安全、生物、化学、核工程)的危险能力,同时不损害良性科学能力,避免概念混淆。
Goodfire researcher Nick Cammarata published a 26,000-word blog post claiming LLMs may be far easier to understand, monitor, and control than assumed, sparking both major excitement and skepticism across the AI community.
研究了perspectivist NLP算法对不同行为标注者(意见分歧或标注稀疏)的预测表现,发现算法更擅长预测倾向于与他人一致的标注者,而标注数量与模型性能无显著正相关。作者还整理了迄今最大的perspectivist序数标注NLP数据集集合。