Alignment: From Pairs to Interactions
How models learn that different signals describe the same content, and why data quality determines the robustness of multimodal alignment.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
1. Image–text pairs: the foundation and its limits
The starting point is conceptually the simplest: image and text pairs that belong together. A photograph with its caption, a product image with its description, a diagram next to its…
2. Beyond the pair: aligning multiple modalities
One of the most important observations from the latest stage of the field is that alignment does not need to remain anchored to image–text to work. ImageBind demonstrated this directly:…
3. Visual instruction: the next level
Image–text pairs (and their extensions to other modalities) train representations, but they do not train the model to follow instructions. For a system to answer "what anomalies are in…
Next step
