From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

Vision-language models like CLIP [1] map images and text into a shared embedding space. This simple objective has turned them into a default building block of modern computer vision, used for zero-shot classification, cross-modal retrieval, and as the perceptual backbone of many generative systems. As they take on this central role, a practical question follows: what has a CLIP model actually learned, and where inside it does that knowledge live?

Mechanistic interpretability answers questions like these by looking past a model's outputs to the computations that produce them, mapping internal components onto functions a human can understand. In language models the program has been fruitful, uncovering linear representations of semantic attributes, attention heads with identifiable roles, and circuits that carry out specific behaviors. Vision-language models have received far less of this scrutiny, and the methods that do exist mostly share one starting point, which, we argue, limits what they can reveal.

Why activations are not enough

The dominant way to interpret CLIP works through activations. The recipe is to run a large image dataset through the model, record the intermediate representations of the component under study, and use those recordings to infer what the component does. TextSpan [2] , one of the first methods to dissect CLIP's vision transformer, is a clear example: it runs ImageNet through each attention head and matches the head's output activations against a large bank of text descriptions, attaching to the head the set of concepts it responds to most.

The recipe is effective, but it inherits the properties of the data it depends on. The most immediate is cost: the dataset has to be collected, curated, and run through the model in full. The more subtle one is bias: the resulting explanation describes the data as much as the model. A head will look "animal-related" only if animals are well represented in the probing set; probe the same head with a different distribution and the description can change. It becomes hard to tell whether an explanation reports a property of the model or an artifact of the dataset.

There is also a question of resolution. The concepts such methods recover are attached to a head as a whole: even when a head is described by several of them, they all sit at the same level, with no handle on the distinct directions inside it. But a head is a linear map containing many such directions, and, as we will see, different directions within a single head can carry unrelated meanings. A head-level description cannot tell them apart.

The key insight: weight space is input-independent

What if, instead of watching a head react to data, we read the head itself? An attention head's behavior is fixed by its weights, learned once during training and unchanged by the input. Interpreting those weights directly would describe what the head is capable of doing — independent of any dataset, and free of the biases a dataset brings.

Not every part of a head lends itself to this, so it helps to recall how multi-head attention is structured. Following Elhage et al. [3] , the output of an attention layer decomposes into a sum of independent per-head contributions, each writing separately into the residual stream:

Each contribution factors into two pieces that play very different roles. The attention matrix $\mathbf{A}^h$ is a data-dependent router: it decides which patches attend to which, and changes with every input. The value-output matrix $\mathbf{W}_{\text{VO}}$ is its opposite: fixed after training, it determines what information the head extracts from the patches it reads and how it transforms that information before writing it back. The router chooses where to look; the VO matrix decides what to do with what it finds. This second, input-independent piece is the one that encodes the head's learned function, and it is what SITH reads.

The VO matrix is a linear map, so we can expose its internal structure with a singular value decomposition [4] :

This yields an ordered set of directions: each right singular vector $\mathbf{v}_i$ is a direction the head writes to the residual stream, each left singular vector $\mathbf{u}_i$ a direction it reads from, and the singular value $\sigma_i$ measures how strongly the head acts along that pair. Ordered by $\sigma$, the leading vectors are the head's dominant reading and writing directions, describing its function in a handful of vectors rather than a full matrix. What remains is to say what those directions mean — the task of the next section. The full pipeline, from the VO matrix of Eq. 1 through the decomposition of Eq. 2 to interpretable concepts, is summarized in Figure 1.

From vectors to concepts: COMP

A singular vector is still only a direction in the residual stream; to interpret it, we have to turn it into words. Because the singular vectors share that space with the $\texttt{[CLS]}$ token, we can project them into CLIP's shared vision-language embedding space, where text lives too. The obvious next move is to read off the nearest text embedding; however, a single nearest neighbor rarely captures a compound meaning. A vector for "red apple" may land closest to "apple" alone, or to "red" alone, losing the combination that defines it.

A better framing is sparse decomposition: describe each singular vector as a small, non-negative combination of concept embeddings that together reconstruct it. Classical sparse-coding algorithms such as Non-Negative Orthogonal Matching Pursuit (NNOMP) do exactly this, but they optimize for one thing only: reconstruction accuracy. At each step they add whichever concept best explains the leftover residual, with no pressure toward a coherent set. A vector can end up explained by "fishing", "urban architecture", and "winter storm" together: numerically faithful, yet meaningless as a description.

Coherent Orthogonal Matching Pursuit (COMP) changes the selection rule to weigh reconstruction against semantic coherence. At each step k, it scores a candidate concept $\hat{\gamma}_i$ as:

The first term is the usual reconstruction criterion; the second rewards concepts that sit close to those already chosen. The weight $\lambda$ sets the balance between them: at $\lambda = 0$, COMP reduces to standard NNOMP, and as $\lambda$ grows it increasingly favors concepts that group into a common theme. The result is a sparse, human-readable explanation for each singular vector — a handful of related concepts rather than a faithful but arbitrary mix.

What the weights encode

Applied to the last four layers of OpenCLIP ViT-L/14, SITH turns each head into a short list of concept sets, one per singular vector. Two patterns stand out.

The first is that individual singular vectors are often interpretable on their own. A single vector usually resolves into a tight group of related concepts — shades of red, words starting with the same letter, kinds of building material — rather than a mix of unrelated ones. Not every vector is this clean: some carry noisy or non-semantic content that no concept set captures well. But among the leading vectors of the final layers, coherent decompositions are the rule rather than the exception.

The decompositions below let you see this directly. For any head and singular vector, the widget shows the concepts COMP assigns and the CC12M [5] images that most strongly excite that direction:

The second pattern appears one level up, across the vectors of a single head. Many heads are specialized: several of their top singular vectors share a theme, so the head as a whole performs a recognizable kind of operation. Head 8 of Layer 23 is a color head, with separate vectors for red, purple, blue, and orange. Head 0 of the same layer keys on materials — leather, metal, paper — and Head 11 on letters, each vector collecting tokens that begin with the same character: weight-space evidence that CLIP has learned to read text in images. Others handle numbers, places, or groupings of people; the widget above lets you browse them all.

These head-level roles line up with what TextSpan [2] finds from activations — reassuring, since the two methods inspect the model in completely different ways. What SITH adds is resolution: where TextSpan describes a head as a whole, SITH attributes each concept to a specific direction, saying which vector inside the head carries which meaning.

This structure is not particular to one model. Applying SITH to other OpenCLIP scales (ViT-B/32, ViT-H/14) and to MobileCLIP — spanning different sizes, architectures, and training recipes — surfaces the same cast of specialized heads: a color head, a place head, a number head, and so on. That models built so differently arrive independently at the same functional parts is, in weight space, an echo of the universality already observed in activation space.

Editing a model without retraining

An interpretation is more convincing when you can act on it. Because SITH works in weight space, its concepts come attached to concrete, editable parts of the model: every concept set belongs to a singular vector with its own singular value. Turn that value down and the concept's contribution to the residual stream shrinks; turn it up and it grows — no fine-tuning, no gradients, no data required.

What changes from task to task is only how we decide which vectors to edit, and by how much. The three results below each make that choice differently.

Suppressing spurious correlations

Waterbirds [6] is built to probe a specific failure: photographs of waterbirds and landbirds are composited onto water and land backgrounds, so that the background becomes a tempting shortcut for predicting the bird. A model that takes the shortcut still does well on the common pairings but falters whenever bird and background disagree. If we could identify the directions in the model that encode background information, we could suppress them and force the model to rely on the bird itself.

To localize those directions, we use an LLM to read the COMP concept set of each singular vector and flag those describing background or location; we then zero the singular values of the flagged vectors. With those directions removed, CLIP's overall accuracy improves from 73.5% to 82.7% and worst-group accuracy from 47.9% to 70.6%, without any labeled data, training, or even access to the Waterbirds distribution. This outperforms TextSpan's [2] head-level ablation (81.8% / 68.0%), which removes entire heads and discards whatever useful information happened to share a head with the spurious features (Table 1). SITH's per-vector suppression removes only the problematic directions.

Removing unsafe concepts

CLIP encodes unsafe content — nudity, violence, and the like — along specific singular vectors, which SITH can locate and neutralize. Here the LLM scores each vector's concept set against the categories of inappropriate content defined by Poppi et al. [7] , and we edit by degree rather than with a single on/off switch. A vector strongly tied to an unsafe category has its singular value set to zero, deleting the direction outright; a vector only moderately related has its singular value flipped to $-1$, which inverts its contribution and actively steers the representation away from the unsafe subspace instead of merely removing it. On the ViSU benchmark [7] , the edited model retrieves safe content more reliably and filters unsafe queries better, with no measurable cost on safe queries (Table 2).

Zero-shot classification

In the two edits above the target was explicit: background features to strip out, unsafe content to push away. Often, though, there is nothing in particular to remove: the goal is simply to do better on a classification task. SITH handles this case too, and without an LLM. Given the task's class names, it scores each singular vector's concept set against them and rescales the singular values in proportion, amplifying the directions relevant to the task and damping the rest. This yields consistent gains across three datasets (Flowers 102 [8] : +1.0 pts, FGVC-Aircraft [9] : +0.3 pts, DTD [10] : +0.8 pts), sharpening the model toward a task without labeled data or gradient updates (Table 3).

Fine-tuning reweights, not rewrites

Fine-tuning a pre-trained model raises a question SITH is well placed to answer: when the weights adapt to a new task, do they preserve the semantic structure they started with, or overwrite it with something task-specific?

To measure the shift, we compute the normalized spectral cosine similarity between the right singular vectors of the pre-trained VO matrix and the fine-tuned one — a value near 1 means the same directions still dominate the head, a value near 0 means they have been replaced. The heatmaps above report it for every head in the last four layers, across three datasets and two adaptation methods.

How much the basis moves depends on the method. Under LoRA the grid is almost uniformly blue: similarity stays around 0.8–0.95, so the head's directions are barely disturbed. Full fine-tuning shifts them more — similarity falls to roughly 0.55–0.70 on average, and lower still on the hardest case, CUB-200 — but it stays well clear of zero, and the change is spread evenly across the layers rather than concentrated in any one. The pre-trained directions are reshaped, not discarded.

Is the part that moves just noise, then? Analyzing the difference matrix $\Delta \mathbf{W} = \mathbf{W}^{\text{ft}}_{\text{VO}} - \mathbf{W}^{\text{pre}}_{\text{VO}}$ with SITH says otherwise. Whatever the update changes, it changes with intent: its leading singular vectors decompose into concepts squarely in the target domain: floral categories for Flowers102 [8] ("alpine flowers", "oriental photinia", "camelia flower"), breeds for Oxford Pets [11] ("english bulldog", "bordeaux mastiff", "egyptian mau"), bird species for CUB-200 [12] ("black catbird", "red legged tinamous", "chiffchaff"). Browse the vocabulary each run wrote into the weights below.

Read together, the two measurements tell one story: fine-tuning works within the semantic basis the model already has, steering its existing directions toward the new task rather than building new ones from scratch.