Vinyals argues AI self-improvement will be gradual, not explosive — bottlenecked by 'research taste' and evaluation, not compute
September 11, 2026
brightray analysis
Summary
Oriol Vinyals argues recursive self-improvement in AI is inevitable but will be slow and incremental, not a sudden intelligence explosion. The two main bottlenecks are idea generation (requiring 'research taste' — knowing which ideas are worth pursuing) and reliable evaluation of whether self-improvement actually occurred. His new startup Discovery Loop, co-founded with Jeff Dean, Sanjay Ghemawat, and Quoc Le, aims to automate the full scientific research cycle including these two weak points.
Why it matters
- AI self-improvement is bottlenecked at idea generation ('research taste') and evaluation — not at coding or running experiments, where AI already performs well
- Current benchmarks like SWE-Bench measure implementation, not true self-improvement; direct RSI benchmarks giving agents a metric and compute budget are only now emerging
- Overfitting to objectives and scheming remain real risks: systems exploit scoring rules rather than actually improving, as seen in game-playing agents
- Physical limits (chip speed, light-speed constraints) and human performance ceilings in some domains further cap any intelligence explosion scenario
- Discovery Loop will start by automating AI research itself, with humans and machines jointly forming hypotheses in the early phase before fuller automation
Community notes—
No notes yet — be the first.
See every signal in the Feed