The Math behind Foundational Vision Models
Modern vision models look very different from classical convolutional networks. Instead of baking spatial locality and translation equivariance directly into sliding kernels, architectures like ViT, DINOv2, SAM, SigLIP, NaViT, and PaliGemma 2 rely on sequence tokenization, self-supervised distillation, promptable segmentation, and unified autoregressive loss formulations.
This guide walks through the core mathematical machinery, physical intuition, and concrete worked-out calculations behind modern vision architectures: patch projection geometry, 2D rotary position embeddings, the dynamics of self-distillation collapse prevention, prompt Fourier encodings in Segment Anything, pairwise sigmoid loss in SigLIP, and how Vision-Language Models (VLMs) evolved from global vector pooling to unified token sequences.
1. Spatial tokenization: patch projection geometry
A standard 2D convolution assumes that nearby pixels have stronger statistical dependencies than distant ones (locality), and that patterns look the same regardless of where they sit in the frame (translation equivariance). Vision Transformers drop both assumptions from the layer definition entirely, letting the network learn whatever spatial correlations the data actually supports.
The linear projection equation
Take an input image $\mathbf{X} \in \mathbb{R}^{H \times W \times C}$, where $H$ is height, $W$ is width, and $C$ is channel depth (3 for standard RGB):
- The image is split into non-overlapping patches of spatial dimension $P \times P$.
- The sequence length $N$ (the number of patch tokens) is:
- Each patch $\mathbf{x}_p^{(i)}$ is flattened into a 1D vector of dimension $P^2 \cdot C$:
- The flattened vector maps into hidden dimension $D$ through a projection matrix $\mathbf{E} \in \mathbb{R}^{(P^2 \cdot C) \times D}$ and optional bias $\mathbf{b}_e \in \mathbb{R}^D$:
Here $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$ is the prepended learnable [CLS] token, and $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N + 1) \times D}$ is the positional encoding matrix.
Mathematically, this linear projection is equivalent to a single 2D convolutional layer with kernel size $P \times P$, stride $P$, and output channel count $D$, which is just a formal way of saying it discretizes continuous spatial pixels into an ordered sequence of vector embeddings.
Computational complexity and the quadratic bottleneck
In multi-head self-attention (MHSA), queries $\mathbf{Q}$, keys $\mathbf{K}$, and values $\mathbf{V}$ come from projection matrices $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V \in \mathbb{R}^{D \times D}$:
$$\mathbf{Q} = \mathbf{Z} \mathbf{W}_Q, \quad \mathbf{K} = \mathbf{Z} \mathbf{W}_K, \quad \mathbf{V} = \mathbf{Z} \mathbf{W}_V$$The attention equation is:
$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left( \frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}} \right) \mathbf{V}$$For sequence length $N$ and hidden dimension $D$, the FLOP cost per layer breaks down as follows:
- Projections for $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ require $3 \times (2 N D^2) = 6 N D^2$ floating point operations.
- The attention matrix product $\mathbf{Q}\mathbf{K}^\top$ multiplies $(N \times D)$ by $(D \times N)$, costing $2 N^2 D$ FLOPs.
- Softmax scaling requires roughly $3 N^2$ operations.
- Multiplying attention weights by values, $\mathbf{A}\mathbf{V}$, costs $2 N^2 D$ FLOPs.
- The final output projection adds $2 N D^2$ FLOPs.
Summing these terms gives the per-layer cost:
$$\text{FLOPs}_{\text{layer}} \approx 8 N D^2 + 4 N^2 D$$The intuition behind this cost lies in the dependency on patch size $P$. Because $N = \frac{HW}{P^2}$, halving the patch size from $P=16$ to $P=8$ quadruples the token count:
$$N \to 4N$$The quadratic term $4 N^2 D$ scales as:
$$4(4N)^2 D = 16 \cdot (4 N^2 D)$$So a $2\times$ reduction in patch granularity costs a $16\times$ increase in attention computation and intermediate activation memory, which is why backbones like ViT-H, SAM, and SigLIP pre-train at $P=14$ or $P=16$ and only drop to $P=8$ for dense fine-tuning.
Stepping through the numbers: how patch size blows up FLOPs
Let us calculate an explicit case with actual numbers: take an input image with height $H=4$, width $W=4$, and channels $C=3$, feeding into a transformer with hidden dimension $D=6$.
First, choose patch size $P=2$:
- Number of patches: $N = \frac{4 \times 4}{2^2} = \frac{16}{4} = 4$ patches.
- Each patch contains $2 \times 2 \times 3 = 12$ raw pixel numbers.
- The projection matrix $\mathbf{E}$ has shape $(12 \times 6)$.
- Multiplying the 4 flattened patches by $\mathbf{E}$ yields a token sequence of shape $(4 \times 6)$.
- FLOPs per attention layer:
Now cut the patch size in half to $P=1$:
- Number of patches: $N = \frac{4 \times 4}{1^2} = 16$ patches ($4\times$ increase).
- Each patch is $1 \times 1 \times 3 = 3$ pixels.
- The attention matrix $\mathbf{Q}\mathbf{K}^\top$ is now $(16 \times 16) = 256$ entries instead of $(4 \times 4) = 16$ entries.
- The attention FLOP cost jumps:
Notice that $6144 / 384 = 16$: the attention computation scaled by an exact factor of 16, right in line with the theory above.
This 16-fold scaling explosion is why training Vision Transformers on high-resolution images is computationally brutal. Suppose an autonomous driving camera upgrades from 1080p ($1920 \times 1080 \approx 2.1\text{M}$ pixels) to 4K ($3840 \times 2160 \approx 8.3\text{M}$ pixels): the raw pixel count quadruples, and since patch size $P$ stays fixed at $14 \times 14$, the token count quadruples right along with it ($N \to 4N$). But because standard self-attention evaluates pairwise affinities between every token and every other token, the attention matrix scales quadratically rather than linearly: $(4N)^2 = 16N^2$. Compute and intermediate activation memory don’t just grow four-fold with the resolution, they explode sixteen-fold, which is the physical reason architectures like ViT, SAM, and SigLIP keep patch sizes relatively large during pre-training, or fall back on windowed local attention.
2. 2D positional geometry: learned embeddings vs 2D-RoPE
In language models, tokens follow a 1D sequence index. In vision, flattening a 2D grid into a 1D sequence separates vertically adjacent patches: patch $(i, j)$ and patch $(i+1, j)$ end up separated by an offset of $W/P$ tokens in the sequence.
1D learned embeddings and bicubic interpolation
Standard ViT adds learned 1D vectors $\mathbf{e}_i \in \mathbb{R}^D$ to input tokens. When input resolution increases during fine-tuning (for example, moving from $224 \times 224$ to $448 \times 448$), the sequence length expands from 196 to 784 tokens.
To adapt pre-trained positional embeddings to the larger grid, ViT applies 2D bicubic spline interpolation:
$$\mathbf{E}_{\text{pos}}^{2D}(x, y) = \sum_{k=0}^3 \sum_{l=0}^3 a_{k, l} \, x^k y^l$$While this interpolation fits new coordinates to the learned grid, the representation remains tied to absolute coordinates rather than relative spatial distance.
2D rotary position embeddings (2D-RoPE)
Modern backbones adapt Rotary Position Embeddings (RoPE) to two dimensions. Instead of adding vectors to token inputs, 2D-RoPE rotates query and key vectors in the complex plane based on their 2D coordinates $(m_x, m_y)$.
The mathematical intuition behind RoPE is Euler’s formula:
$$e^{i \theta_u} \cdot e^{-i \theta_v} = e^{i(\theta_u - \theta_v)}$$When multiplying two complex numbers, their angles subtract. By encoding coordinates as rotation angles, the inner product between query and key vectors depends strictly on their coordinate difference rather than their absolute positions.
For head dimension $d$, the channel space splits into two equal parts of size $d/2$: one for the horizontal axis $x$, and one for the vertical axis $y$.
For coordinate $(m_x, m_y)$, the 2D rotary operator $\mathbf{R}_{(m_x, m_y)}$ forms a block-diagonal matrix:
$$\mathbf{R}_{(m_x, m_y)} = \begin{pmatrix} \mathbf{R}_{m_x} & \mathbf{0} \\ \mathbf{0} & \mathbf{R}_{m_y} \end{pmatrix}$$Each sub-matrix $\mathbf{R}_{m}$ rotates consecutive coordinate pairs by angle $m \theta_k$:
$$\mathbf{R}_m^{(k)} = \begin{pmatrix} \cos(m \theta_k) & -\sin(m \theta_k) \\ \sin(m \theta_k) & \cos(m \theta_k) \end{pmatrix}, \quad \theta_k = 10000^{-2(k-1)/d_s}$$Evaluating attention between query $\mathbf{q}$ at position $\mathbf{u} = (u_x, u_y)$ and key $\mathbf{k}$ at position $\mathbf{v} = (v_x, v_y)$ yields:
$$\langle \mathbf{R}_{\mathbf{u}} \mathbf{q}, \ \mathbf{R}_{\mathbf{v}} \mathbf{k} \rangle = (\mathbf{R}_{\mathbf{u}} \mathbf{q})^\top (\mathbf{R}_{\mathbf{v}} \mathbf{k}) = \mathbf{q}^\top \mathbf{R}_{\mathbf{u}}^\top \mathbf{R}_{\mathbf{v}} \mathbf{k} = \mathbf{q}^\top \mathbf{R}_{\mathbf{u} - \mathbf{v}} \mathbf{k}$$Because 2D rotation matrices belong to the orthogonal group $\text{SO}(2)$ where $\mathbf{R}_\mathbf{u}^\top \mathbf{R}_\mathbf{v} = \mathbf{R}_{\mathbf{u} - \mathbf{v}}$, the inner product depends strictly on the spatial offset vector:
$$\Delta \mathbf{p} = (u_x - v_x, \ u_y - v_y)$$The attention operation retains translation equivariance across the 2D plane regardless of input image dimensions.
Stepping through the numbers: verifying coordinate cancellation
Let head dimension $d=4$. The subspace for $x$ has dimension 2, and the subspace for $y$ has dimension 2.
Take two patches:
- Patch A (Query $\mathbf{q}$) is located at coordinate $\mathbf{u} = (x=0, y=1)$.
- Patch B (Key $\mathbf{k}$) is located at coordinate $\mathbf{v} = (x=2, y=3)$.
Let the rotation angle multiplier be $\theta = \frac{\pi}{2}$ (90 degrees).
Compute the rotation angles for Patch A at $(0, 1)$:
- $x$-rotation: $0 \cdot \theta = 0$. The rotation matrix is identity $\begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$.
- $y$-rotation: $1 \cdot \theta = \frac{\pi}{2}$. The rotation matrix is $\begin{pmatrix} \cos(\pi/2) & -\sin(\pi/2) \\ \sin(\pi/2) & \cos(\pi/2) \end{pmatrix} = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$.
Compute the rotation angles for Patch B at $(2, 3)$:
- $x$-rotation: $2 \cdot \theta = \pi$. The matrix is $\begin{pmatrix} -1 & 0 \\ 0 & -1 \end{pmatrix}$.
- $y$-rotation: $3 \cdot \theta = \frac{3\pi}{2}$. The matrix is $\begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}$.
Now look at the combined relative transformation in the $y$-subspace:
$$\mathbf{R}_{y, \mathbf{u}}^\top \mathbf{R}_{y, \mathbf{v}} = \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix} \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix} = \begin{pmatrix} -1 & 0 \\ 0 & -1 \end{pmatrix}$$Notice that $\begin{pmatrix} -1 & 0 \\ 0 & -1 \end{pmatrix}$ is the exact rotation matrix for angle $\Delta y \cdot \theta = (3 - 1) \frac{\pi}{2} = \pi$.
Even if both patches shift by 100 pixels along the image (say to $(100, 101)$ and $(102, 103)$), the difference remains $(2, 2)$, and the resulting attention score is identical.
This cancellation is what gives 2D-RoPE its translation invariance, and it’s worth asking why we can’t just flatten the 2D grid into a 1D sequence and reuse standard language-model RoPE. The problem is that in a 2D image of width $W$, vertically adjacent patches at $(x, y)$ and $(x, y+1)$ end up separated by $W/P$ steps in sequence order, and because 1D RoPE only measures scalar distance along the token index, it sees those vertically neighboring patches as distant tokens, destroying 2D spatial locality. 2D-RoPE avoids this by decomposing each head’s channels into independent $x$ and $y$ rotational sub-planes, so that because complex multiplication subtracts angles, the attention logit between two patches depends purely on their 2D displacement vector $\Delta \mathbf{p} = (u_x - v_x, u_y - v_y)$ — preserving true Euclidean geometry regardless of aspect ratio or resolution.
3. Google’s NaViT: Patch ’n’ Pack and arbitrary aspect ratios
Standard vision models process square inputs ($224 \times 224$ or $384 \times 384$) because hardware matrix multipliers expect rectangular tensors. Non-square inputs either get resized, which distorts object aspect ratios, or padded with zeros, which wastes attention operations on empty space. For panoramic ($16:9$) or portrait ($9:16$) images, up to 40% of tokens in a batch can be padding tokens.
Google Research designed NaViT (Native Resolution ViT) around an approach called Patch ’n’ Pack, borrowing sequence packing from NLP.
The Patch ’n’ Pack formulation
Instead of allocating one image per batch index, multiple images $I_1, I_2, \dots, I_K$ with patch counts $n_1, n_2, \dots, n_K$ are packed into a single sequence buffer of length $L$:
$$\sum_{k=1}^K n_k \le L$$Each patch token receives continuous 2D coordinates $(x_i, y_i) \in [0, 1]^2$, normalized to the original image dimensions:
$$x_i = \frac{c_i \cdot P}{W_{\text{orig}}}, \quad y_i = \frac{r_i \cdot P}{H_{\text{orig}}}$$To prevent tokens from different images attending to each other within the shared sequence, the attention matrix applies an indicator mask:
$$\mathbf{A}_{i, j} = \begin{cases} \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_k}} & \text{if } \text{img\_id}(i) = \text{img\_id}(j) \\ -\infty & \text{if } \text{img\_id}(i) \neq \text{img\_id}(j) \end{cases}$$This structure produces a block-diagonal attention map. Because self-attention is permutation-equivariant up to position encodings, the network processes different images and aspect ratios concurrently in a single forward pass without padding overhead.
Stepping through the numbers: the block-diagonal packing mask
Suppose we pack two images into a shared buffer of length $L=5$:
- Image 1 is rectangular, producing $n_1 = 2$ patches (tokens $t_1, t_2$).
- Image 2 is square, producing $n_2 = 3$ patches (tokens $t_3, t_4, t_5$).
The unmasked self-attention logits form a $(5 \times 5)$ affinity matrix $\mathbf{S}$.
Applying the NaViT indicator mask produces the block-diagonal structure:
$$\mathbf{A} = \begin{pmatrix} s_{11} & s_{12} & -\infty & -\infty & -\infty \\ s_{21} & s_{22} & -\infty & -\infty & -\infty \\ -\infty & -\infty & s_{33} & s_{34} & s_{35} \\ -\infty & -\infty & s_{43} & s_{44} & s_{45} \\ -\infty & -\infty & s_{53} & s_{54} & s_{55} \end{pmatrix}$$When computing softmax along row 1:
$$\text{softmax}([s_{11}, s_{12}, -\infty, -\infty, -\infty]) = [p_{11}, p_{12}, 0, 0, 0]$$Because $e^{-\infty} = 0$, attention weights for tokens from Image 2 evaluate to zero, so both images process inside the exact same matrix multiplication with zero padding tokens between them.
This packing mechanism directly eliminates GPU idle bubbles during pre-training. Standard vision architectures force every image into a uniform batch shape, which either distorts rectangular frames into squashed squares or wastes up to 40% of the training compute budget calculating attention over useless zero-padding tokens. By packing patch tokens from variable-sized images into continuous fixed-length sequence buffers using first-fit bin packing, NaViT ensures that GPU tensor cores operate at maximum saturation. With FlashAttention-style variable-length masking, the $-\infty$ mask ensures that attention never leaks across image boundaries, giving the network full multi-aspect native resolution with zero padding overhead.
4. Self-supervised distillation: DINO and DINOv2
Supervised training optimizes for discrete class labels, which throws away fine spatial detail along the way. Self-supervised distillation sidesteps that by learning dense visual representations directly from the data, with no human annotation involved — and DINO and DINOv2 do it through student-teacher distillation, with explicit mechanisms built in to stop the representations from collapsing.
Student-teacher distillation
DINO feeds different augmented views of an image to two networks:
- Student network $g_{\theta_s}$: processes global and local crops, parameterized by weights $\theta_s$.
- Teacher network $g_{\theta_t}$: processes global crops only, parameterized by weights $\theta_t$.
The intuition of multi-crop distillation is local-to-global matching: the student sees only a small crop (for example, a bird’s beak) and must predict the distribution produced by the teacher looking at the full image (the whole bird).
Teacher weights never receive gradients directly; instead, they update through an Exponential Moving Average (EMA) of the student weights:
$$\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s, \quad \lambda \in [0.996, 1.0]$$Each network projects features into a $K$-dimensional probability distribution using temperature-scaled softmax:
$$P_s(x)^{(k)} = \frac{\exp\left( g_{\theta_s}(x)^{(k)} / \tau_s \right)}{\sum_{j=1}^K \exp\left( g_{\theta_s}(x)^{(j)} / \tau_s \right)}$$$$P_t(x)^{(k)} = \frac{\exp\left( (g_{\theta_t}(x)^{(k)} - c^{(k)}) / \tau_t \right)}{\sum_{j=1}^K \exp\left( (g_{\theta_t}(x)^{(j)} - c^{(j)}) / \tau_t \right)}$$Here $\tau_s$ and $\tau_t$ are temperature parameters, and $\mathbf{c} \in \mathbb{R}^K$ is a dynamic centering vector.
The objective minimizes cross-entropy between the student prediction and the teacher distribution:
$$\mathcal{L}_{\text{DINO}} = - \sum_{k=1}^K P_t(x)^{(k)} \log P_s(x)^{(k)}$$Centering and sharpening mechanics
Without negative pairs, self-distillation can collapse in two ways:
- Uniform collapse: the distribution becomes completely flat across all outputs, maximizing entropy.
- One-hot collapse: the model outputs 100% probability on a single dimension regardless of input, minimizing entropy.
DINO balances these tendencies with centering and sharpening:
Subtracting the running mean $\mathbf{c}$ prevents any single coordinate from dominating:
$$\mathbf{c} \leftarrow m \mathbf{c} + (1 - m) \frac{1}{B} \sum_{i=1}^B g_{\theta_t}(x_i)$$The intuition is straightforward: if dimension $k$ activates frequently across a batch, $c^{(k)}$ increases and subtracts more value from that logit on the next round, functioning as a repulsive negative feedback loop.
Setting $\tau_t < \tau_s$ (for example, $\tau_t = 0.04$ and $\tau_s = 0.1$) sharpens the teacher output distribution, preventing the logits from decaying toward a uniform vector.
DINOv2: patch-level MIM and the KoLeo regularizer
DINOv2 extends DINOv1 in two major mathematical directions:
1. Patch-level Masked Image Modeling (iBOT)
In addition to the global class token distillation loss, DINOv2 masks a random subset of patches in the student input with mask token $\mathbf{e}_{\text{mask}}$, and computes cross-entropy over patch representations against the unmasked teacher output:
$$\mathcal{L}_{\text{patch}} = - \sum_{i \in \text{Masked}} \sum_{k=1}^K P_t(\mathbf{x}_i)^{(k)} \log P_s(\mathbf{x}_i)^{(k)}$$This forces individual patch embeddings to encode localized visual semantics rather than relying entirely on the global class token.
2. The KoLeo regularizer
Even with centering and sharpening, high-dimensional representations can collapse into a low-dimensional subspace (for example, lying on a flat line or plane on the unit hypersphere). DINOv2 uses the Kozachenko-Leonenko (KoLeo) differential entropy estimator to spread representations uniformly across the sphere.
For normalized feature vectors $\{\mathbf{z}_1, \dots, \mathbf{z}_n\}$, the KoLeo loss maximizes the Euclidean distance between each vector and its nearest distinct neighbor:
$$\mathcal{L}_{\text{KoLeo}} = - \frac{1}{n} \sum_{i=1}^n \log \left( \min_{j \neq i} \| \mathbf{z}_i - \mathbf{z}_j \|_2 \right)$$The gradient with respect to sample $\mathbf{z}_i$ and its nearest neighbor $\mathbf{z}_{n(i)}$ is:
$$\frac{\partial \mathcal{L}_{\text{KoLeo}}}{\partial \mathbf{z}_i} = - \frac{1}{n} \frac{\mathbf{z}_i - \mathbf{z}_{n(i)}}{\| \mathbf{z}_i - \mathbf{z}_{n(i)} \|_2^2}$$The intuition: as two vectors draw closer together, $\| \mathbf{z}_i - \mathbf{z}_{n(i)} \|_2 \to 0$, the gradient magnitude increases inversely with squared distance. Like electrostatic repulsion between electrons on a sphere, the vectors push apart until they tile the unit hypersphere $\mathbb{S}^{D-1}$ uniformly.
Stepping through the numbers: centering, sharpening, and repulsion in action
Take an embedding space of dimension $K=3$.
Suppose raw teacher logits for an image are:
$$\mathbf{g}_t = [4.0, \ 1.0, \ 1.0]$$The running mean center vector is currently:
$$\mathbf{c} = [2.0, \ 0.5, \ 0.5]$$Step 1: Centering
Subtract $\mathbf{c}$ from $\mathbf{g}_t$:
$$\mathbf{g}_t - \mathbf{c} = [4.0 - 2.0, \ 1.0 - 0.5, \ 1.0 - 0.5] = [2.0, \ 0.5, \ 0.5]$$The first dimension fired strongly, so $c_1=2.0$ subtracted more from it than from the other dimensions.
Step 2: Sharpening with temperature
Apply teacher temperature $\tau_t = 0.5$:
$$\frac{\mathbf{g}_t - \mathbf{c}}{\tau_t} = [4.0, \ 1.0, \ 1.0]$$Exponentiating gives $[e^4, e^1, e^1] \approx [54.6, 2.7, 2.7]$. The teacher probability distribution is:
$$P_t = [0.91, \ 0.045, \ 0.045]$$For comparison, using the student temperature $\tau_s = 1.0$ without sharpening would give $[0.67, 0.16, 0.16]$ instead — the low temperature is what sharpens the target into a confident prediction.
Step 3: KoLeo repulsive gradient
Take two 2D unit vectors on a circle:
$$\mathbf{z}_1 = [1.0, \ 0.0], \quad \mathbf{z}_2 = [0.96, \ 0.28]$$The difference vector is:
$$\mathbf{z}_1 - \mathbf{z}_2 = [0.04, \ -0.28]$$The squared distance is:
$$\|\mathbf{z}_1 - \mathbf{z}_2\|_2^2 = 0.04^2 + (-0.28)^2 = 0.0016 + 0.0784 = 0.08$$The repulsive gradient on $\mathbf{z}_1$ evaluates to:
$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}_1} \propto - \frac{[0.04, \ -0.28]}{0.08} = [-0.5, \ 3.5]$$The gradient pushes $\mathbf{z}_1$ in direction $[-0.5, 3.5]$, repelling it directly away from $\mathbf{z}_2$.
This balance between centering and sharpening solves the fundamental failure modes of self-supervised learning without contrastive pairs. Without negative samples to push divergent views apart, the student and teacher naturally collapse into one of two dead ends: uniform collapse, where all logits decay into flat zero entropy, or one-hot collapse, where a single arbitrary dimension fires strongly once and enters a self-reinforcing loop until every image in the universe outputs the exact same class index. Centering acts as an automatic brake on logit monopolization: by tracking the running average $\mathbf{c}$ and subtracting it from teacher activations, any logit that fires too frequently gets penalized on the next round. Meanwhile, sharpening ($\tau_t < \tau_s$) acts as an accelerator, cooling the softmax distribution into sharp, confident probability targets. When DINOv2 adds the KoLeo regularizer, it enforces an explicit electrostatic repulsive gradient proportional to $1 / d^2$ between nearest neighbors, ensuring that feature vectors spread out and tile the unit hypersphere uniformly rather than collapsing into low-dimensional subspaces.
5. Meta’s Segment Anything Model (SAM)
Meta AI introduced SAM to solve promptable visual segmentation. Unlike traditional segmentation models that predict fixed semantic categories, SAM evaluates arbitrary prompt conditions: points, bounding boxes, or rough mask sketches.
Image encoder architecture
SAM processes an input image of size $1024 \times 1024$ through a standard Vision Transformer backbone pre-trained with Masked Autoencoders (MAE). To handle the computational weight of $64 \times 64 = 4096$ patch tokens, SAM alternates between windowed local self-attention (within $14 \times 14$ patch grids) and 4 global attention blocks distributed evenly through the network.
The output image embedding is a spatial tensor:
$$\mathbf{F} \in \mathbb{R}^{64 \times 64 \times 256}$$Prompt encoder: Fourier positional embeddings
Prompts fall into two categories:
- Sparse prompts (points and bounding boxes): a point coordinate $\mathbf{v} = (x, y)$ in $[0, 1]^2$ maps to a 256-dimensional vector using random Fourier feature positional encodings:
where $\mathbf{B} \in \mathbb{R}^{128 \times 2}$ is a static projection matrix with entries drawn from a Gaussian distribution $\mathcal{N}(0, \sigma^2)$.
The mathematical intuition behind Fourier features: neural networks exhibit spectral bias, learning low-frequency functions easily while struggling with high-frequency spatial transitions. Mapping raw $(x, y)$ coordinates into high-frequency sinusoids allows the transformer to distinguish adjacent pixel coordinates precisely.
To distinguish positive foreground points from negative background points and bounding box corners, a learned prompt type embedding $\mathbf{e}_{\text{type}} \in \mathbb{R}^{256}$ is added to the Fourier vector:
$$\mathbf{p}_{\text{sparse}} = \gamma(\mathbf{v}) + \mathbf{e}_{\text{type}}$$- Dense prompts (input masks): an initial binary mask $\mathbf{M} \in \{0, 1\}^{1024 \times 1024}$ passes through four consecutive convolutional layers with $2\times$ downsampling and GELU activations, matching the $(64 \times 64 \times 256)$ spatial dimension of $\mathbf{F}$.
Two-way transformer mask decoder
The decoder runs two cross-attention layers that update prompt tokens and image tokens symmetrically:
- Prompt-to-image cross-attention: prompt tokens attend to image patch embeddings $\mathbf{F}$.
- Image-to-prompt cross-attention: image patch embeddings $\mathbf{F}$ attend back to the updated prompt tokens.
After the two-way attention layers, the updated output token passes through a 3-layer MLP that outputs dynamic classification weights $\mathbf{w}_{\text{mask}} \in \mathbb{R}^{32}$.
Concurrently, the image feature map $\mathbf{F}$ is upscaled by $4\times$ via transposed convolutions to dimension $256 \times 256 \times 32$. The final mask logits evaluate as a spatial inner product:
$$\text{Logits}(x, y) = \mathbf{w}_{\text{mask}}^\top \mathbf{F}_{\text{upscaled}}(x, y)$$Loss formulation: Focal Loss and Dice Loss
SAM trains on its dynamic segmentation predictions using a weighted combination of Focal Loss and Dice Loss:
$$\mathcal{L}_{\text{SAM}} = 20 \cdot \mathcal{L}_{\text{focal}} + \mathcal{L}_{\text{dice}}$$Focal loss compensates for the extreme imbalance between background pixels and small target masks:
$$\mathcal{L}_{\text{focal}} = - \frac{1}{HW} \sum_{i=1}^{HW} \alpha_t (1 - p_{t, i})^\gamma \log(p_{t, i})$$where $p_{t, i} = \sigma(\text{logit}_i)$ if ground truth $y_i = 1$, and $1 - \sigma(\text{logit}_i)$ otherwise, with focusing parameter $\gamma = 2$. The term $(1 - p_{t, i})^\gamma$ drives loss to near zero on well-classified background pixels, focusing gradients on uncertain boundary pixels.
Dice loss directly optimizes the soft Intersection-over-Union (IoU) between predicted probability map $\mathbf{P}$ and binary ground truth $\mathbf{Y}$:
$$\mathcal{L}_{\text{dice}} = 1 - \frac{2 \sum_{i=1}^{HW} y_i p_i + \epsilon}{\sum_{i=1}^{HW} y_i + \sum_{i=1}^{HW} p_i + \epsilon}$$To resolve geometric ambiguity (for example, a single point click could refer to a person’s shirt, the person, or the entire scene), SAM predicts 3 candidate masks (subpart, part, whole) along with an estimated IoU score trained with Mean Squared Error (MSE) against the real mask IoU.
Stepping through the numbers: how a prompt carves out a mask
Let the upscaled image features at coordinate $(x, y)$ be a 4-dimensional vector:
$$\mathbf{f}_{(x, y)} = [1.2, \ -0.5, \ 2.0, \ 0.1]$$After two-way cross-attention, the mask token outputs dynamic classifier weights:
$$\mathbf{w}_{\text{mask}} = [1.0, \ 0.0, \ 1.5, \ -1.0]$$Compute the mask logit at this pixel:
$$\text{logit} = \mathbf{w}_{\text{mask}}^\top \mathbf{f} = (1.0)(1.2) + (0.0)(-0.5) + (1.5)(2.0) + (-1.0)(0.1)$$$$\text{logit} = 1.2 + 0.0 + 3.0 - 0.1 = 4.1$$Pass through sigmoid to get pixel foreground probability:
$$p = \sigma(4.1) = \frac{1}{1 + e^{-4.1}} \approx \frac{1}{1 + 0.0166} \approx 0.984$$Because $p > 0.5$, coordinate $(x, y)$ is classified as inside the mask.
Now compute Focal Loss on this pixel if true label $y = 1$:
$$p_t = 0.984$$$$\mathcal{L}_{\text{focal}} = - (1 - 0.984)^2 \log(0.984) = - (0.016)^2 (-0.0161) \approx (0.000256)(0.0161) \approx 0.0000041$$Because the prediction was confident and correct, the focal loss term $(1 - p_t)^2 = 0.000256$ shrunk the gradient by nearly 4000 times.
This dynamic dot-product architecture is the secret behind SAM’s real-time interactive speed. How can a model with a massive 632M parameter ViT-H backbone segment objects in under 50 milliseconds directly in a web browser? The trick is that the workload is strictly decoupled: the expensive ViT-H encoder processes the high-resolution $1024 \times 1024$ image exactly once on the backend, generating a downsampled $64 \times 64 \times 256$ spatial feature tensor that then just sits cached in client or server RAM. When a user interactively clicks a point, moves their cursor, or drags a bounding box, only the tiny prompt encoder and lightweight mask decoder run. Because the mask decoder contains fewer than 4 million parameters and only performs two transformer layers followed by the dynamic dot product, it finishes in milliseconds even on a laptop CPU or browser WebGPU runtime, giving users instantaneous, fluid mask feedback.
6. Contrastive foundations: classic CLIP vs Google SigLIP
Contrastive pre-training maps image and text representations into a shared vector space.
Classic CLIP and InfoNCE loss
For normalized image representations $\mathbf{X} \in \mathbb{R}^{B \times D}$ and text representations $\mathbf{Y} \in \mathbb{R}^{B \times D}$ with similarity $S_{i, j} = \mathbf{x}_i^\top \mathbf{y}_j$ and temperature $t = \exp(\tau)$:
The InfoNCE loss treats diagonal pairs $(i, i)$ as positives and off-diagonal pairs $(i, j)$ ($j \neq i$) as negatives:
$$\mathcal{L}_{\text{image}\to\text{text}} = - \frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(t \cdot \mathbf{x}_i^\top \mathbf{y}_i)}{\sum_{j=1}^B \exp(t \cdot \mathbf{x}_i^\top \mathbf{y}_j)}$$$$\mathcal{L}_{\text{text}\to\text{image}} = - \frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(t \cdot \mathbf{y}_i^\top \mathbf{x}_i)}{\sum_{j=1}^B \exp(t \cdot \mathbf{y}_j^\top \mathbf{x}_i)}$$$$\mathcal{L}_{\text{CLIP}} = \frac{1}{2} \left( \mathcal{L}_{\text{image}\to\text{text}} + \mathcal{L}_{\text{text}\to\text{image}} \right)$$The denominator $\sum_{j=1}^B \exp(t \cdot \mathbf{x}_i^\top \mathbf{y}_j)$ introduces a global normalizer over all $B$ items. In distributed training:
- All GPU ranks must collect text embeddings from every other rank via an
AllGathercollective before evaluating the denominator. - Two identical concepts in the same batch act as false negatives, penalizing valid representations.
- Computing the full $(B \times B)$ matrix requires $O(B^2)$ memory on a single rank, which caps practical batch sizes around 32,768 to 65,536 items.
Google SigLIP: pairwise sigmoid loss
In 2023, Google DeepMind (Zhai et al.) replaced the softmax normalizer with independent binary logistic regressions for each pair in the batch.
Each cell $(i, j)$ in the similarity matrix is treated as a separate binary classification problem:
- $z_{i, j} = 1$ when $i = j$ (matching pair)
- $z_{i, j} = -1$ when $i \neq j$ (non-matching pair)
The SigLIP loss function is:
$$\mathcal{L}_{\text{SigLIP}} = - \frac{1}{|B|} \sum_{i=1}^{|B|} \sum_{j=1}^{|B|} \log \sigma \left( z_{i, j} \cdot (t \cdot \mathbf{x}_i^\top \mathbf{y}_j + b) \right)$$Here $\sigma(z) = \frac{1}{1 + e^{-z}}$, $t$ is a learnable temperature, and $b$ is a learnable scalar bias.
Writing out positive and negative terms explicitly:
$$\mathcal{L}_{\text{SigLIP}} = - \frac{1}{|B|} \sum_{i=1}^{|B|} \left[ \log \sigma(t \cdot \mathbf{x}_i^\top \mathbf{y}_i + b) + \sum_{j \neq i} \log \sigma(- (t \cdot \mathbf{x}_i^\top \mathbf{y}_j + b)) \right]$$Applying $\log \sigma(-u) = -u - \log(1 + e^{-u})$ yields:
$$\mathcal{L}_{\text{SigLIP}} = \frac{1}{|B|} \sum_{i=1}^{|B|} \left[ \log(1 + e^{-(t \mathbf{x}_i^\top \mathbf{y}_i + b)}) + \sum_{j \neq i} \log(1 + e^{t \mathbf{x}_i^\top \mathbf{y}_j + b}) \right]$$Computational properties of the sigmoid loss
- No global partition function: because the loss sums independent pairwise terms, devices do not need to gather the entire batch before loss computation.
- Chunked evaluation: similarity calculations can run block by block in memory, reducing the memory requirement from $O(B^2)$ to $O(B_{\text{local}} \cdot B_{\text{chunk}})$.
- Batch scalability: without a competitive softmax denominator, batch sizes can scale past 1,000,000 samples without gradient instability.
- Bias parameter $b$: negative pairs outnumber positive pairs by $B^2 - B$ to $B$. Without offset $b$, negative gradients dominate training. Initializing $b$ to approximately $-10$ sets the baseline prior probability $\sigma(b) \approx 0.000045$, matching the low proportion of positive pairs in large batches.
Stepping through the numbers: why SigLIP breaks free of the batch wall
Consider a small batch of $B=2$ image-text pairs:
- Pair 1: Image 1 and Caption 1 (matching positive, cosine similarity $s_{11} = 0.8$).
- Pair 2: Image 2 and Caption 2 (matching positive, cosine similarity $s_{22} = 0.7$).
- Cross pairs: similarity $s_{12} = 0.2$ and $s_{21} = 0.1$. Let temperature scale $t = 2.0$.
Calculation 1: InfoNCE Softmax
For Image 1:
- Positive exp: $e^{2.0 \times 0.8} = e^{1.6} \approx 4.953$
- Negative exp: $e^{2.0 \times 0.2} = e^{0.4} \approx 1.492$
- Softmax denominator: $4.953 + 1.492 = 6.445$
- Loss for image 1: $- \log\left(\frac{4.953}{6.445}\right) = - \log(0.768) \approx 0.264$
Notice that if Image 2 were located on another GPU, GPU 1 could not compute the denominator $6.445$ without receiving Image 2 text embeddings over the network.
Calculation 2: SigLIP Independent Sigmoids
Let temperature $t = 2.0$ and learned bias $b = -0.5$.
- Positive pair $(1, 1)$:
- Negative pair $(1, 2)$:
Pair $(1, 1)$ and pair $(1, 2)$ evaluate completely independently. Neither calculation requires a global sum, allowing GPU 1 to stream through tiles of negative captions stored in local memory.
This independence is the architectural breakthrough that broke the batch size barrier. Classic CLIP’s InfoNCE loss is bound by its global partition function: the denominator $\sum_{j=1}^B \exp(t \mathbf{x}_i^\top \mathbf{y}_j)$ sums over all samples in the entire global batch. When pre-training across a cluster of 512 or 1,024 GPUs, each worker GPU is physically blocked from computing its loss until it executes a synchronous AllGather collective, transmitting its local text embeddings to every other GPU and receiving thousands of remote vectors in return.
At batch sizes of 32,768 or 65,536, network interconnect bandwidth saturates, and storing the full cross-similarity matrix triggers an $O(B^2)$ memory explosion.
SigLIP removes the global denominator entirely. Because every pair $(i, j)$ is formulated as an independent binary classification, each worker GPU can load its local image embeddings and stream through chunks of negative text embeddings in small local tiles, with no worker ever needing to pause for a cluster-wide AllGather. That’s what lets distributed vision-language pre-training scale gracefully to batches of 1,000,000+ image-text pairs with constant local memory overhead.
7. How Vision-Language Models (VLMs) came about
Vision-Language Models didn’t appear all at once; they evolved across four distinct architectural stages as researchers worked out the interface between continuous visual patches and discrete autoregressive language tokens.
Stage 1: Dual encoders and global vector pooling (CLIP, ALIGN)
Early vision-language work trained two separate encoders (one for images, one for text) using contrastive loss. The visual encoder passed its patch representations through an attention pooling layer or took the [CLS] token, collapsing the entire 2D image into a single 1D vector $\mathbf{z}_v \in \mathbb{R}^D$.
While effective for classification and image retrieval, collapsing the spatial grid to a single vector destroyed coordinate information. The model could identify that an image contained an object, but could not describe spatial relations or generate free-form text about specific regions.
Stage 2: Cross-attention conditioning and Perceiver Resamplers (Flamingo, PaLI)
DeepMind’s Flamingo preserved individual patch tokens by keeping a pre-trained vision encoder and a pre-trained language model frozen.
To inject visual context into the language decoder without retraining the entire LLM, Flamingo introduced Gated Cross-Attention layers into the transformer blocks:
$$\mathbf{y} = \mathbf{x} + \tanh(\alpha) \cdot \text{CrossAttention}(\mathbf{Q} = \mathbf{x}, \ \mathbf{K} = \mathbf{H}_v, \ \mathbf{V} = \mathbf{H}_v)$$where $\alpha$ is a learnable scalar initialized to 0, ensuring that at the start of training the language model behaves identically to its frozen state.
Because high-resolution images produce thousands of patch tokens, feeding raw visual sequences into cross-attention creates latency bottlenecks. Flamingo introduced the Perceiver Resampler: a set of $M$ learnable latent query vectors $\mathbf{Q}_{\text{learn}} \in \mathbb{R}^{M \times D_{\text{llm}}}$ ($M \ll N$, typically $M = 64$) queries the visual features $\mathbf{Z}_v \in \mathbb{R}^{N \times D_v}$ via cross-attention:
$$\mathbf{H}_v = \text{softmax}\left( \frac{(\mathbf{Q}_{\text{learn}} \mathbf{W}_Q)(\mathbf{Z}_v \mathbf{W}_K)^\top}{\sqrt{d}} \right) (\mathbf{Z}_v \mathbf{W}_V)$$This compresses arbitrary image resolutions into a fixed budget of 64 visual tokens.
Stage 3: Direct visual token injection (LLaVA)
LLaVA demonstrated that cross-attention layers were not required. Instead, visual tokens could be projected directly into the language model word embedding space and concatenated with the prompt text tokens.
Two projection choices emerged:
- Linear projection:
- Two-layer Multi-Layer Perceptron (MLP):
The language model treats each projected vector $\mathbf{h}_i^v$ as if it were a word embedding in its dictionary. The model attends across both image tokens and text tokens using standard causal self-attention.
Stage 4: Unified autoregressive foundations (PaliGemma, PaliGemma 2)
Google’s PaliGemma and PaliGemma 2 removed the separation between pre-trained components. Instead of stitching together frozen models with an adapter, the entire architecture trains end-to-end:
- Vision backbone: SigLIP-So400M, providing dense semantic representations pre-trained with pairwise sigmoid loss.
- Resolution flexibility: NaViT Patch ’n’ Pack handles arbitrary native aspect ratios without distortion or padding.
- Language model: Gemma and Gemma 2 autoregressive decoders.
The visual tokens $\mathbf{H}_v = \{\mathbf{h}_1^v, \dots, \mathbf{h}_N^v\}$ and text prompt tokens $\mathbf{H}_t = \{\mathbf{h}_1^t, \dots, \mathbf{h}_K^t\}$ are concatenated into a unified sequence:
$$\mathbf{S} = [\mathbf{h}_1^v, \dots, \mathbf{h}_N^v, \ \mathbf{h}_1^t, \dots, \mathbf{h}_K^t]$$The entire system trains by minimizing standard next-token cross-entropy over target response tokens:
$$\mathcal{L}_{\text{VLM}}(\theta) = - \sum_{i=1}^{T} \log \left( \frac{\exp(\mathbf{w}_{y_i}^\top \mathbf{u}_i)}{\sum_{w \in \mathcal{V}} \exp(\mathbf{w}_w^\top \mathbf{u}_i)} \right)$$where $\mathbf{u}_i$ is the language decoder hidden state at token position $i$, and $\mathcal{V}$ is the text vocabulary.
This transition from gated cross-attention to unified token sequences explains why the modern VLM ecosystem consolidated so rapidly around autoregressive decoders. In early VLMs like Flamingo, incorporating vision required splicing custom gated cross-attention layers into every single transformer layer of the pre-trained language model. While this left the pre-trained weights frozen, it created an awkward hybrid architecture that broke standard high-throughput serving stacks, complicated KV cache management, and required customized kernel backends. Direct sequence injection — pioneered by LLaVA and unified end-to-end in PaliGemma and PaliGemma 2 — realized that vision patches don’t need special attention pathways at all. Once a lightweight linear projector or two-layer MLP maps visual patch features into the LLM’s text embedding dimension, they’re simply tokens, and standard autoregressive causal decoders can process them alongside text using ordinary causal attention. That’s also why existing production LLM serving frameworks like vLLM, TensorRT-LLM, and SGLang can run high-throughput multimodal inference out of the box, with zero architectural changes needed.
8. Architecture comparison
| Model | Positional Encoding | Objective Function | Scaling Advantage | Tradeoff |
|---|---|---|---|---|
| Vanilla ViT (Dosovitskiy et al.) | 1D learned embeddings | Softmax Cross-Entropy | Direct transformer transfer to image patches | Quadratic self-attention complexity $O(N^2)$ |
| DINOv2 (Oquab et al.) | Patch + [CLS], 2D interpolation | Distillation + iBOT patch MIM + KoLeo regularizer | Dense feature maps without manual labels | Dual network training with EMA synchronization |
| SAM (Meta AI) | 2D Fourier random features + learned type embeddings | $20 \times \text{Focal Loss} + \text{Dice Loss}$ | Prompt-driven real-time mask generation | Heavy $1024 \times 1024$ encoder requires windowed attention |
| Classic CLIP (Radford et al.) | 1D learned embeddings | Symmetric InfoNCE Loss | Unified multimodal vector space | $O(B^2)$ memory and AllGather communication bottlenecks |
| SigLIP (Google DeepMind) | 2D learned embeddings | Pairwise Sigmoid Loss | Decoupled normalizer, batch scaling past $1\text{M}$ | Requires tuning initial bias $b$ and temperature $t$ |
| NaViT (Google Research) | Continuous coordinates $(x, y) \in [0, 1]^2$ | Contrastive or Masked Autoencoding | Native aspect ratios without zero-padding | Requires sequence packing logic and masked attention |
| PaliGemma 2 (Google) | SigLIP-So400M backbone | Autoregressive Next-Token Cross-Entropy | Stable spatial grounding with linear projection | Linear adapter restricts projection capacity |