LLM Lab

Multimodal LLMs

To let a language model see, you have to turn pixels into something it can read. There are two main ways.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Cross-attention: add a window onto the image

Flamingo keeps the language model frozen. An image encoder produces many features; a Perceiver Resampler squeezes them into a few fixed-size visual tokens using learned queries. New gated cross-attention layers between the LLM’s blocks let text tokens read those visual tokens; a tanh gate, initialised at 0, controls how much vision flows in.

Visual tokens: treat the image as words

PaLM-E-style models project image features into the LLM’s embedding space and put them in the input sequence beside the text tokens. The same idea carries other signals, such as robot states, and RT-2 goes one step further by outputting robot actions as tokens.

Flamingo
image→vision encoder→Perceiver Resampler → 5 visual tokens→gated cross-attention inside a frozen LLM→text
PaLM-E
image→vision encoder→project to embeddings→visual tokens mixed with text tokens→LLM → text or actions
Two ways to give an LLM eyes. Flamingo adds new cross-attention layers that read visual tokens; PaLM-E feeds visual tokens straight into the sequence the LLM already reads.

A worked example

An image of any size goes through a vision encoder and comes out as, say, 32 feature vectors. Flamingo’s resampler has 5 learned queries; attention from those queries over the 32 features gives a 5 × 32 map and 5 output vectors, the visual tokens. Gated cross-attention layers between the frozen LLM’s blocks let the text read those 5 tokens. Because the gate starts at tanh(0) = 0, on the first training step the model is exactly the original language model.

From images to actions

Treating other signals as tokens generalises well. PaLM-E feeds images and robot state into the LLM; RT-2 writes robot actions as tokens in the output, so one model goes from a camera image and an instruction to motor commands. Vision-language models built this way can describe unusual scenes, such as a chair falling off a truck on a highway, in plain language.

Easy to confuse

Check yourself · 1 of 3

In Flamingo, which part of the model stays frozen?

Check yourself · 2 of 3

Why does Flamingo’s gate start at zero?

Check yourself · 3 of 3

What is the idea behind PaLM-E and similar models?

Read more in the books

  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 9, multimodal large language models.
  • AI Engineering, Chip Huyen: Ch. 2, the section on multimodal models.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section IV, multi-modal LLMs.
All books
Labtrace an image
How to read this
  • Pick a route. Flamingo: the dotted grid is the Perceiver Resampler’s attention map, 5 learned queries (rows) over 32 image features (columns).
  • Visual tokens: the striped squares are image tokens placed in the same sequence as the word chips.
  • The numbered list walks through the route; for Flamingo, the gate slider shows how much visual signal tanh(α) lets in.