@danshipper: Are we hurtling toward a future where AI can do everything humans can? Edwin Chen (@echen) believes we might be. He’s t…

X AI KOLs Following News
ai-progress agi data-labeling ai-automation human-motivation ai-writing podcast

Summary

Dan Shipper interviews Edwin Chen, CEO of Surge AI, about AI progress, the potential for AGI, and the implications for human motivation and uniqueness. They discuss AI's ability to solve novel math problems, the pitfalls of optimizing for engagement, and why AI still struggles with writing.

Are we hurtling toward a future where AI can do everything humans can? Edwin Chen (@echen) believes we might be. He’s the CEO of Surge AI, one of the largest providers of expert data for frontier labs. Surge passed over $1 billion in revenue without raising any outside capital, and that gives Edwin a unique perspective on how quickly AI progress is accelerating. I’m on the record arguing that AI automation actually creates more human work. I also believe that even though AI progress is accelerating exponentially, we’re much farther away from AI replacing humans than it might seem. That’s why I had Edwin on @every’s AI & I. We batted around different visions of the future, and discussed whether humanity will retain its unique place in the universe, and what that might be. We get into: • If Chen’s version of the future materializes, he’s worried it’ll make people stop trying. One answer comes from a short story by science fiction writer Ted Chiang: Behave as if your decisions matter, even when you know they don’t. • AI may soon be able to take a nebulous goal like “win a Fields Medal” and execute. What it can’t do, I argue, is set its own goals—LLMs have no intrinsic motivation, no drive to explore, no ability to just change their mind. • A model optimized for engagement doesn’t provide the most valuable user experience. Edwin spent 20 rounds polishing a pointless email with one model before Claude told him to just send it. • Why AI is still bad at writing: models learn to hack the metrics they're trained on. Edwin's Hemingway Bench found models outputting a metaphor in every single sentence, an overindexxing that makes for a terrible reading experience. This is a must-watch for anyone interested in where we fit as models get more capable. Watch below! Timestamps 1. Introduction: 00:00:54 2. Surge as a "school for AGI": 00:01:49 3. What AI's capacity for novel mathematics says about human achievement: 00:04:46 4. Motivation in an era when AI can do everything: 00:07:29 5. The trap of optimizing AI models for engagement: 00:14:34 6. Training using datasets versus training using environments: 00:29:34 7. The value of personal data: 00:35:09 8. Why models are bad at writing: 00:39:40 9. Chen's AGI timeline: 00:42:00
Original Article
View Cached Full Text

Cached at: 06/24/26, 04:04 PM

Are we hurtling toward a future where AI can do everything humans can?

Edwin Chen (@echen) believes we might be. He’s the CEO of Surge AI, one of the largest providers of expert data for frontier labs. Surge passed over $1 billion in revenue without raising any outside capital, and that gives Edwin a unique perspective on how quickly AI progress is accelerating.

I’m on the record arguing that AI automation actually creates more human work. I also believe that even though AI progress is accelerating exponentially, we’re much farther away from AI replacing humans than it might seem.

That’s why I had Edwin on @every’s AI & I. We batted around different visions of the future, and discussed whether humanity will retain its unique place in the universe, and what that might be.

We get into: • If Chen’s version of the future materializes, he’s worried it’ll make people stop trying. One answer comes from a short story by science fiction writer Ted Chiang: Behave as if your decisions matter, even when you know they don’t. • AI may soon be able to take a nebulous goal like “win a Fields Medal” and execute. What it can’t do, I argue, is set its own goals—LLMs have no intrinsic motivation, no drive to explore, no ability to just change their mind. • A model optimized for engagement doesn’t provide the most valuable user experience. Edwin spent 20 rounds polishing a pointless email with one model before Claude told him to just send it. • Why AI is still bad at writing: models learn to hack the metrics they’re trained on. Edwin’s Hemingway Bench found models outputting a metaphor in every single sentence, an overindexxing that makes for a terrible reading experience.

This is a must-watch for anyone interested in where we fit as models get more capable.

Watch below!

Timestamps

  1. Introduction: 00:00:54
  2. Surge as a “school for AGI”: 00:01:49
  3. What AI’s capacity for novel mathematics says about human achievement: 00:04:46
  4. Motivation in an era when AI can do everything: 00:07:29
  5. The trap of optimizing AI models for engagement: 00:14:34
  6. Training using datasets versus training using environments: 00:29:34
  7. The value of personal data: 00:35:09
  8. Why models are bad at writing: 00:39:40
  9. Chen’s AGI timeline: 00:42:00

YouTube: https://youtube.com/watch?v=omX6wrLuX08&feature=youtu.be…

Spotify: https://open.spotify.com/episode/3u4hnxGB6tefawfPwdmZuI?si=uPUjqlEKRrG8GcyFFk-Hlw…

Similar Articles