ai-methodology

The author revised the success criterion for coding-agent evaluations by moving from requiring three perfect 88/88 runs to a criterion where each case must pass at least 2 of 3 runs, reducing randomness impact and improving stability.

This paper critiques standard attention metric methods in transformers, revealing that choices like keeping or dropping sink tokens can reverse conclusions, and proposes using compositional data analysis to separate sink and content attention components for more accurate interpretation.

This paper identifies the 'evaluation trap' where AI benchmarks inadvertently stabilize dominant paradigms by narrowing what counts as progress, and introduces Epistematics, a meta-evaluative methodology to ensure evaluation criteria discriminate true capability from proxy behaviors.

25-year-old podcast host Dwarkesh Patel has interviewed key figures from top AI labs including OpenAI, Anthropic, and DeepMind, such as Karpathy, Hassabis, Dario Amodei, and Ilya Sutskever. He publicly shared his AI-assisted "one-week preparation" workflow: having AI列出必读资料, tracking gaps in understanding, using AI to map out the full landscape, and implementing the code himself. Time magazine included him in the "AI 100" list for 2024.