The Real Problem of Multimodality
What it means to integrate text, images, audio and other modalities, and how perception, alignment, reasoning, generation and action fit together.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
1. A modality is not only an input type
We call a modality a distinct way of encoding information about the world. Text is a discrete sequence of symbols, an image has spatial structure, audio adds temporal continuity, tone,…
2. The problem is not adding modalities, but crossing them without destroying them
A system can accept an image and still remain deeply text-centric: it only has to turn the image into a caption and do everything else over that caption.
3. The shared space matters, but it is not the only possible architecture
The most common version of an introduction to multimodality treats the shared representation space almost as the essence of the entire field.
Next step
