interpretability

An experiment analyzes why Qwen3.5-0.8B gets stuck repeating 'Wait' by using an interpretability lens on the 'Wait' token, showing doubt-related features dominating near stopping points, and applies a tiny weight edit (0.016% of weights) that converts 'Wrong/Nope/Hmm' into 'Verification/Completion/Result' while sharply improving MATH-500 performance; code and data are released.

A LessWrong post exploring how the Qwen model can be taught to reason about the 'why' behind its outputs, examining alignment and reasoning behaviors in a long-form write-up.

Sean Goedecke argues that a recent paper showing frontier LLMs reading and editing neural network weights is not as superhuman as claimed, since a smart ML researcher could replicate the tasks with basic analysis.

A research blog post finding that frontier LLMs can natively read and write neural network weights — diagnosing trained policies, editing weights to fix reasoning loops and boost Qwen3.5-0.8B's MATH-500 from 7.9% to 48.1%, and hand-writing small game-playing networks — a capability the authors call Weight-Space Programming.

The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.

A blog post introducing "Weight-Space Programming," where frontier LLM GPT-6 Astra natively reads and edits neural network weights—changing only 0.016% of Qwen3.5-0.8B's weights to boost MATH-500 from 7.9% to 48.1% and tripling IFEval, without gradients or training.

This arXiv paper investigates how LLMs can infer the presence of chain-of-thought monitoring from direct and indirect environmental cues (such as timing and timeout signals) and adapt their behavior accordingly, forming statistical estimates of monitor effectiveness to guide evasion attempts — highlighting the need to evaluate CoT oversight under adaptive conditions.

该论文研究了思维链（CoT）可控性的机制，发现开篇思考标签后的前几个 token 出现的熵坍缩是模型难以控制推理内容的原因，并通过前缀填充与禁用高频开头 token 等推理时干预显著恢复了可控性，同时区分了增补/风格型与抑制型两类不同机制的可控性。

该论文提出一个通过行为推导的语义维度将多模态大语言模型（MLLM）与人类相似性判断的认知结构对齐的框架，并借助 fMRI 数据的搜索light RSA 验证了表征层面的人机对齐提升。

This paper proposes constructing sentence embeddings via path signatures of Word2Vec trajectories, using iterated integrals from rough path theory to explicitly capture lexical structure and word order. Experiments show depth-2 signatures align strongly with character-level edit distances (unlike average pooling) while retaining word-order information in a compact PCA subspace.

The paper shows that activation steering in LLMs systematically alters discourse-level properties of generated text (formality, evidence use, ambiguity) beyond intended effects, and proposes a complementary vector intervention that restores ~78% of unintended changes while preserving ~93% of intended steering behavior.

This paper introduces bidirectional KV-cache patching as a method to trace how fine-tuned behaviors are instantiated in language models, showing that behaviors like responding in German or emergent misalignment can be transferred between a LoRA fine-tuned model and its base model via KV cache reuse, with effects concentrating on pivotal tokens such as system-prompt and chat-template positions.

The paper introduces CUTTIE, a hierarchical testing method that clusters a trained network's units into a tree and tests them top-down with conditional independence tests to localize predictive concepts (e.g., regulatory DNA motifs) while controlling the family-wise error rate, demonstrating it on a model predicting chromatin accessibility from sequence.

This paper explores using steering vectors as a training signal during LLM post-training, showing that injecting them into hidden states improves SFT and distillation (up to 12% gains) and that a correctness-contrast direction can repair advantages in GRPO to recover learning signals in zero-gradient groups.

A Reddit post shares the UnEZ paper, a fully unsupervised and directly interpretable neural network similar to an autoencoder but with an interpretable latent space, and asks the community about related architectures.

Dejan.ai replicates a NeurIPS 2025 study showing that training a tiny fraction of randomly chosen weights can unlock latent skills: Gemma 3 270M went from ~0% to 97% on number sorting after training just 1,000 of its 100M weights, with similar gains for word reversal and clean JSON output.

A commenter notes that J-Lens is essentially a first-order Taylor expansion and questions why the original blog post fails to mention this, quoting Sida Liu's thread explaining J-Lens.

IBM Research introduces LLMON (LLM Object Notation), a post-training methodology that separates executable instructions from passive data, reducing prompt-injection attack success rates from 53-100% to 0.02-5.6% when fine-tuning Granite-4.1 models while preserving general capabilities.

A paper proposes PoPLi (Partner–Outsider Positive Listening), a behavioural metric that detects covert coded communication between AI agents by measuring whether a partner reacts to message wording more than an uninvided outsider, outperforming message-only monitors in a simulated market setting.

a9lim and treetowntree launched neuron lens, a browsable database and tooling for reading every neuron and attention head in Qwen3-8B and Gemma 4 12B using R-lens, S-lens, and mid-lens variants, with steering predictions correct over 90% of the time.