Multimodal System Architectures
Four multimodal architecture families, their differences in quality, cost and latency, and when each way of combining modalities makes sense.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
1. Visual encoder + connector + language model
The most widespread approach in recent years consists of three chained components: a visual encoder that processes the image and produces a high-dimensional representation, a connection…
2. Fusion through cross-attention
Flamingo, published by DeepMind in 2022, introduced a different approach: instead of processing the image before the text and passing its representation as input, it inserted…
3. Native multimodal tokenization
The third approach is the most radical: instead of connecting a visual encoder to a language model through some type of connector, the system discretizes images or audio into tokens of the…
Next step
