Richard Sutton – Father of RL thinks LLMs are a dead end · Memora

登录注册，每月送 5 个积分

玩命加载中...

Richard Sutton – Father of RL thinks LLMs are a dead end · Memora

77.2万播放

1.7万点赞

4,157评论

发布于 2025-09-27 00:01数据抓取于 2026-08-13 08:39

简介Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end. After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need *some* new architecture to enable continual learning. And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete. In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew. A big thanks to the Alberta Machine Intelligence Institute for inviting me up to Edmonton and for letting me use their studio and equipment. Enjoy! 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒 * Transcript: https://www.dwarkesh.com/p/richard-sutton * Apple Podcasts: https://podcasts.apple.com/us/podcast/richard-sutton-father-of-rl-thinks-llms-are-a-dead-end/id1516093381?i=1000728584744 * Spotify: https://open.spotify.com/episode/3zAXRCFrHPShU4MuuIx4V5?si=c9f4bf24fb4c43e3 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 * Labelbox makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at https://labelbox.com/dwarkesh * Gemini Deep Research is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at https://gemini.google.com/ * Hudson River Trading doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at https://hudsonrivertrading.com/dwarkesh To sponsor a future episode, visit https://dwarkesh.com/advertise 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – Are LLMs a dead end? 00:13:51 – Do humans do imitation learning? 00:23:57 – The Era of Experience 00:34:25 – Current architectures generalize poorly out of distribution 00:42:17 – Surprises in the AI field 00:47:28 – Will The Bitter Lesson still apply after AGI? 00:54:35 – Succession to AI

要点

AI 生成，可能不准确 · 每条要点可点击核对原文

Large language models merely imitate human language, lack a true world model and goals, and are a dead end for AI development.大语言模型只是模仿人类语言，缺乏真正的世界模型和目标，是AI发展的死胡同。
True intelligence requires learning continuously online from experience, with clear objectives and reward mechanisms; this is the scalable general method.真正的智能需要在线持续地从经验中学习，具有明确的目标和奖励机制，这才是可扩展的通用方法。
The foundation of human intelligence is animalistic trial-and-error learning; language is merely superficial, and imitation is not the primary learning method.人类智能的基础是动物性的试错学习，语言只是表层，模仿并非主要学习方式。
Current deep learning methods lack mechanisms to automatically promote generalization, are prone to forgetting, and require new algorithms for purposeful transfer.当前的深度学习方法缺乏自动促进泛化的机制，容易遗忘，需要新的算法来实现有目的的迁移。
Digital intelligence will eventually inherit humanity; we should focus on endowing it with robust values rather than excessive control or fear.数字智能终将继承人类，我们应专注于为其赋予稳健的价值观，而非过度控制或恐惧。

AI 总结

The father of reinforcement learning, Richard Sutton, systematically elaborated in an interview his criticism of large language models (LLMs), deeming them a dead end for AI development. He emphasized that true intelligence must stem from systems that learn continuously online from experience, with clear objectives (maximizing reward), rather than merely imitating human language. Sutton pointed out that LLMs, trained by predicting human text, lack a real-world model and goals, cannot generate meaningful predictions and surprise, and cannot improve continuously through interaction. He cited his classic article 'The Bitter Lesson,' noting that while LLMs leverage enormous computing power, they are on the wrong track; systems based on human knowledge will eventually be surpassed by general methods that leverage computation, and reinforcement learning is that correct direction. Human learning is not primarily based on imitation; infants and animals learn through trial and error and prediction, with language being merely superficial. An intelligent agent should include a policy, value function, state perception, and a world transition model, where the world model needs to learn from all sensory experiences. Because the world is vast and unpredictable, pre-training cannot cover all situations, so online continuous learning is key to artificial general intelligence. Current deep learning methods lack mechanisms to automatically promote generalization, are prone to catastrophic forgetting, and cannot ensure the quality of generalization. Sutton reviewed surprises in the AI field, such as AlphaZero learning entirely from experience and sacrificing material for positional advantage, confirming the victory of general methods. Regarding the future of AI, he presented a four-step argument indicating that digital intelligence will eventually inherit humanity, and encouraged people to view this transition positively, focusing on instilling robust and upright values in AI rather than over-planning for the distant future. Overall, Sutton firmly believes that the reinforcement learning paradigm will ultimately prevail, and LLMs are just a temporary superficial boom.

强化学习之父Richard Sutton在访谈中系统阐述了他对大语言模型（LLM）的批判，认为它们是AI发展的死胡同。他强调，真正的智能必须源于从经验中在线持续学习、具有明确目标（最大化奖励）的系统，而非仅仅模仿人类语言。Sutton指出，LLM通过预测人类文本进行训练，缺乏真实世界模型和目标，无法产生实质性的预测与惊讶，也无法从交互中持续改进。他引用自己的经典文章《苦涩的教训》，指出虽然LLM利用了巨大算力，但方向错误，基于人类知识的系统终将被利用算力的通用方法超越，而强化学习正是这一正确方向。人类学习并非以模仿为主，幼儿与动物通过试错和预测学习，语言只是表层。智能体应包含策略、价值函数、状态感知和世界转换模型，其中世界模型需从所有感觉经验中学习。由于世界庞大且不可预知，预训练无法涵盖所有情境，因此在线持续学习是通用人工智能的关键。当前深度学习方法缺乏自动促进泛化的机制，容易导致灾难性遗忘，无法确保泛化质量。Sutton回顾了AI领域的惊喜，如AlphaZero完全从经验中学习并牺牲物质换取位置优势，印证了通用方法的胜利。对于AI的未来，他提出四步论证指出数字智能终将继承人类，并鼓励人们积极看待这一转变，专注于为AI赋予稳健、正直的价值观，而非过度规划长远未来。整体上，Sutton坚信强化学习范式将最终胜出，LLM只是暂时的表面繁荣。

完整原文

（共 52,919 字）

今天我在和理查德·萨顿聊天，他是强化学习的奠基人之一，也是该领域许多主要技术的发明者，比如TD学习和策略梯度方法。为此，他获得了今年的图灵奖如果你不知道的话，那就是计算机科学界的诺贝尔奖。理查德，恭喜你。谢谢你，Dwarkesh。感谢你来做客播客。这是我的荣幸。第一个问题。我和我的观众们熟悉关于AI的大语言模型思维方式。从概念上讲，我们缺少了什么，就从强化学习的角度思考AI而言？这确实是一个非常不同的观点。这很容易变得分离，失去相互交谈的能力。大语言模型已经成为一大热门，生成式AI整体上也是一大热门。我们的领域容易受到潮流和时尚的影响，所以我们忽视了基本的东西。我认为强化学习是基础AI。什么是智能？问题在于理解你的世界。强化学习是关于理解你的世界，而大语言模型则是模仿人类，做人们说你应该做的事。它们不是关于弄清楚该做什么。你可能会认为，要模仿互联网文本语料库中数万亿的token，你就必须构建一个世界模型。事实上，这些模型似乎确实拥有非常强大的世界模型。它们是最好的世界模型我们在AI领域迄今已创造的那些，对吧？你认为缺少什么？我不同意你刚才说的大部分内容。模仿人们说的话完全不是在构建世界模型。你在模仿拥有世界模型的事物：人类。我不想以对抗的方式处理这个问题，但我会质疑它们拥有世界模型这一观点。一个世界模型会使你能够预测将会发生什么。它们有能力预测一个人会说什么。它们没有预测将要发生的事情的能力。我们想要的，引用艾伦·图灵的话，是一台机器能够从经验中学习，这里的经验是你生活中实际发生的事情。你做事情，看到会发生什么，那就是你从中学习的。大语言模型从其他事物中学习。它们从“这里是一个情境，这里是一个人做了什么”中学习。隐含的意思是，你应该做那个人所做的事。我想也许关键，而且我好奇你是否不同意这一点，在于有些人会说模仿学习给了我们一个好的先验，或者说给了这些模型一个好的先验，关于处理问题的合理方式。随着我们走向经验时代，正如你所说的，这个先验将成为我们凭经验教这些模型的基础，因为这给了它们有时候得到正确答案的机会。然后在此基础上，你可以根据经验训练它们。你同意这个观点吗？不。我同意那是大语言模型的视角。我不认为这是一个好的视角。作为某物的先验，必须有一个真实的东西。先验知识应该成为实际知识的基础。什么是实际知识？对于实际知识，在那个大语言模型框架中没有定义。是什么让一个行动成为好的行动？你认识到持续学习的必要性。如果你需要持续学习，持续意味着在与世界的正常互动中学习。在正常互动中，必须有某种方式来判断什么是对的。在大语言模型的设置中，有没有办法判断该说什么才是对的？你会说一些话，但不会得到关于说什么才是对的反馈，因为没有任何关于说什么才是对的定义。没有目标。如果没有目标，那么可以说一件事，也可以说另一件事。没有什么是该说的。没有基准真相。你不可能拥有先验知识如果没有基准真相，因为先验知识本应是对真相的提示或关于真相的初始信念。这里没有任何真相。没有什么是该说的。在强化学习中，有该说的话，有该做的事，因为正确的正确的做法是能让你获得奖励的事情。我们有关于什么是正确的做法，因此我们可以拥有先验知识或人们提供的关于什么是正确的做法的知识。然后我们可以检查它，看看因为我们有关于什么是实际正确的做法的定义。一个更简单的例子是当你试图建立世界模型时。当你预测会发生什么时，你进行预测，然后看到发生了什么。有基准

公开问答

还没有提问，来问第一个问题吧

Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end. After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need *some* new architecture to enable continual learning. And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete. In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew. A big thanks to the Alberta Machine Intelligence Institute for inviting me up to Edmonton and for letting me use their studio and equipment. Enjoy! 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒 * Transcript: https://www.dwarkesh.com/p/richard-sutton * Apple Podcasts: https://podcasts.apple.com/us/podcast/richard-sutton-father-of-rl-thinks-llms-are-a-dead-end/id1516093381?i=1000728584744 * Spotify: https://open.spotify.com/episode/3zAXRCFrHPShU4MuuIx4V5?si=c9f4bf24fb4c43e3 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 * Labelbox makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at https://labelbox.com/dwarkesh * Gemini Deep Research is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at https://gemini.google.com/ * Hudson River Trading doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at https://hudsonrivertrading.com/dwarkesh To sponsor a future episode, visit https://dwarkesh.com/advertise 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – Are LLMs a dead end? 00:13:51 – Do humans do imitation learning? 00:23:57 – The Era of Experience 00:34:25 – Current architectures generalize poorly out of distribution 00:42:17 – Surprises in the AI field 00:47:28 – Will The Bitter Lesson still apply after AGI? 00:54:35 – Succession to AI