Why Transformers Work: See Heidegger, Not Descartes
AI researchers have inherited more from Descartes than they might realize. The Cartesian worldview divides reality into two realms, the external world of objects and the internal realm of ideas. In this picture, language is simply a mapping device, a way to attach words to ideas in the mind, which themselves represent objects out there in the world. This representational picture has shaped centuries of thought in philosophy, linguistics, and cognitive science. It implies that for genuine intelligence to arise a system must build internal models of the external world, and that language would be secondary, little more than a label pasted onto those models. If that were true then training a system on text alone, without access to perception or embodiment, should not get us very far. At best it would produce a…
