Dwarkesh Patel
Long-form AI interviewer & podcaster
Dwarkesh Podcast
on dwarkesh.com

John Schulman, Beren Millidge, and Charlie O'Neill argue RSI faces hard limits: AI can 10x well-specified research but cannot think its way to new training paradigms.

September 11, 2026
brightray analysis
Summary

Three researchers identify key blockers to recursive self-improvement: AI can accelerate research dramatically when objectives are well-specified, but cannot discover new paradigms because 'thinking alone can't generate new bits.' The likely ceiling is not a hard wall but a series of discontinuities — like the jump from pre-training to RL — each requiring a breakthrough current models may not be able to generate. John Schulman argues the last durable human role is defining what we actually want, i.e., alignment and objective specification.

Why it matters
  • AI can plausibly deliver ~10x research speedup for well-specified objectives, but thinking alone cannot generate new bits needed to invent new paradigms — a hard epistemological ceiling.
  • The most credible RSI failure mode is not a hard wall but an asymptote just below the threshold needed to dominate human R&D, requiring a discontinuity the current paradigm cannot self-discover.
  • Schulman identifies objective specification and alignment — deciding what we actually want — as the last durable human job, even after AI automates all technical execution.
  • Beren Millidge flags that self-directed objective proposal (AIs setting their own research agenda) may exhibit a Moravec's Paradox: easy for humans, surprisingly hard for AI.
  • Charlie O'Neill cites the Kaplan scaling law annealing error as a concrete example: an AI thinking carefully about existing data could have caught it years earlier, cutting real progress time.
View original on dwarkesh.com

Community notes

No notes yet — be the first.


See every signal in the Feed