Multimodal LLMs
To let a language model see, you have to turn pixels into something it can read. There are two main ways.
Cross-attention: add a window onto the image
Flamingo keeps the language model frozen. An image encoder produces many features; a Perceiver Resampler squeezes them into a few fixed-size visual tokens using learned queries. New gated cross-attention layers between the LLM’s blocks let text tokens read those visual tokens; a tanh gate, initialised at 0, controls how much vision flows in.
Visual tokens: treat the image as words
PaLM-E-style models project image features into the LLM’s embedding space and put them in the input sequence beside the text tokens. The same idea carries other signals, such as robot states, and RT-2 goes one step further by outputting robot actions as tokens.
A worked example
An image of any size goes through a vision encoder and comes out as, say, 32 feature vectors. Flamingo’s resampler has 5 learned queries; attention from those queries over the 32 features gives a 5 × 32 map and 5 output vectors, the visual tokens. Gated cross-attention layers between the frozen LLM’s blocks let the text read those 5 tokens. Because the gate starts at tanh(0) = 0, on the first training step the model is exactly the original language model.
From images to actions
Treating other signals as tokens generalises well. PaLM-E feeds images and robot state into the LLM; RT-2 writes robot actions as tokens in the output, so one model goes from a camera image and an instruction to motor commands. Vision-language models built this way can describe unusual scenes, such as a chair falling off a truck on a highway, in plain language.
Easy to confuse
- “Visual tokens” are vectors, not words: they never pass through the text tokenizer.
- Freezing the LLM keeps its language skills but limits how deeply vision and text are mixed; models that train everything together can mix more but risk losing some text ability.
In Flamingo, which part of the model stays frozen?
Flamingo keeps the pretrained language model frozen and adds new cross-attention layers between its blocks, so language ability is preserved while vision is added.
Why does Flamingo’s gate start at zero?
The visual output is multiplied by tanh of a learned gate. At 0 the gate adds nothing, so training starts from the unchanged language model.
What is the idea behind PaLM-E and similar models?
An image encoder produces vectors that are projected into the LLM’s embedding space and placed in the sequence like words. The LLM attends to them with its normal attention.
Read more in the books
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 9, multimodal large language models.
- AI Engineering, Chip Huyen: Ch. 2, the section on multimodal models.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section IV, multi-modal LLMs.