Beyond the Transformer
How the field is trying to move beyond pure Transformer scaling by combining tools, search, inference-time memory, world models and robotics.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
1. Why the Transformer is no longer a complete map
The Transformer reorganized the field because it was parallelizable, scalable and extremely general. But scaling it also made several limits increasingly visible.
1.1 Truth, uncertainty and hallucination
Another important limitation appears here. Generative LLMs trained around next-token prediction are not directly optimized to distinguish truth, falsehood and unknown information. They are…
2. From next-token prediction to search over solution spaces
One of the most important directions in this new phase is a renewed emphasis on something that the LLM boom had pushed somewhat into the background: search.
Next step
