Skip to main content

Insights

Before the answer: what the Jacobian Lens can reveal

From an ASCII face to a concept swap, a closer look at intermediate representations—and the evidence needed to connect a readable word to behavior.

7 min read
On this page

A face made of characters has a nose consisting of a single ^. The prompt never names it. Yet in an example Anthropic presents using Sonnet 4.5, the Jacobian Lens reads out nose at an intermediate layer at that position.

We can already ask a model what it sees and read its explanation. This example changes the observation point: before the word becomes an answer, researchers can find a related representation inside the network.

The eye and nose positions of an ASCII face, with J-Lens word readouts at three layers
Figure 1. Redrawn from the ASCII-face panel of Gurnee et al., Figure 3, preserving two character positions, three layer labels, and their words. L54, L71, and L83 use the paper's normalized depth labels, not raw layer indices. These are the authors' readouts, not a QingBo replication or words already emitted by the model.

There is still a distance between reading out a word and knowing why a model answered as it did. A word can be a useful clue or an invitation to overinterpret. The question worth following through the Jacobian Lens paper is how the reading is obtained—and what additional evidence connects it to behavior.

Before a word, a computation

A chat interface makes reasoning look like a sequence of text: steps first, answer later. But producing any token requires computation across the network's layers. The resulting text then becomes context for later computation. An implementation may cache and reuse states from earlier positions.

A chain of thought can therefore be both evidence and working material. It can retain intermediate results without being a complete execution record.

This concern has experimental grounding. A 2023 study of chain-of-thought faithfulness modified reasoning text and found that reliance on it varied by model and task. A 2025 follow-up found that, in the tested multiple-choice and hint settings, models could use a hint without acknowledging it in their reasoning.

These findings do not make reasoning traces useless. They sharpen the question: what might we miss if we examine only the words a model produces?

J-Lens adds an internal observation channel. It reads an intermediate state rather than requesting another explanation. Turning those numbers into recognizable words, however, still requires a measurement procedure.

Getting the coordinates right

A transformer maintains a high-dimensional state at each position, commonly called the residual stream. Layers read from it and write their results back. At the final layer, the output head maps the state into vocabulary scores.

The straightforward approach, the Logit Lens, applies that same head to intermediate states. Those states still have transformations ahead of them; their representation may not yet suit the final output head. An unreadable result can reflect a mismatch in the reading method rather than an absence of information.

The Jacobian Lens inserts a map between the two. A Jacobian describes a local relationship: if a state changes slightly, how does the remaining network propagate that change? Averaging over texts, source positions, and current and later target positions gives a linear map from each layer to the final-layer representation. To read a state, the lens applies that map, then the model's normalization and output head.

Fitting averages local effects across texts and positions; reading applies a per-layer map and output head to an intermediate vector
Figure 2. Method diagram based on Methods and the reference implementation. Top: fitting per-layer maps. Bottom: reading one state. Cells and bars are illustrative, without measured values. Dashed lines show local effects or the direct-read path. Current and future mean target positions within the supplied text, not an unwritten response.

The reference implementation makes the two stages concrete. Fitting requires internal model access and gradient computation. Applying an existing lens maps an activation into a readout. The repository says the paper fitted lenses on 1,000 sequences of 128 tokens and notes that this is unoptimized reference code. That is an experimental setup, not a universal minimum or a quality guarantee.

Averaging matters. An effect in one context may reflect how a concept happens to be used there. Pooling contexts aims to recover a more general disposition toward verbalization. The result remains an approximation, rather than an exact replay of all the downstream computation.

Consequently, a word's rank is not the probability that the model is thinking about it. Even after softmax, this is a vocabulary readout from a lens, not a calibrated probability of a mental state.

A reading is a starting point

Finding nose in the face example gives us a meaningful candidate direction. It does not establish that the model relies on that direction when it subsequently answers. A tool might expose something readable that is incidental to the behavior being explained.

The paper's sport experiment goes further. The model is asked to think of a sport and name it in one word. In the illustrated case, both lens readout and answer point to Soccer. The authors then remove its projection along the corresponding lens direction and add an equal-magnitude projection along Rugby. Continuing the forward pass changes the reported sport to Rugby.

The original paper's sport swap, rank correlations across fourteen categories, and candidate output ranks before and after intervention
Figure 3. Gurnee et al., original Figure 6, retained in English. Left: the sport swap. Upper right: lens-versus-output rank correlation for 10 candidate answers across 14 categories. Lower right: target output rank before and after swaps, restricted to targets initially ranked at least 11th. Layer labels are normalized as in the paper. All experiments are the authors', not QingBo tests.

The two plots on the right answer different questions. The correlation plot asks whether candidates ranked highly by the lens also rank highly in the model's output. The swap plot asks whether changing the corresponding direction moves an initially unlikely target upward. The latter adds an intervention, offering stronger causal evidence than a matching readout alone.

Its conclusion still concerns particular directions under particular conditions. It does not locate all the model's knowledge of football in one vector or guarantee predictable effects from arbitrary edits. Assessing an explanation also means asking about failed interventions, alternative positions, intervention strengths, and control directions.

This is the research habit we would preserve: do not substitute legibility for strength of evidence. A familiar noun makes understanding feel close. The experiment still has work to do.

What makes a workspace?

The paper calls a collection of verbalizable representations the J-space and examines whether they can be modulated, used for intermediate reasoning, and used by different downstream computations. That motivates the comparison with a global workspace: information is available for multiple operations, rather than merely stored.

The word space needs care. Vocabulary-indexed lens directions are neither mutually orthogonal nor a unique set of concept coordinates. Without limiting how many directions may participate, an activation admits many decompositions. The authors therefore study sparse, nonnegative combinations. They do not identify the entire residual stream with a region called consciousness.

Capacity and variance figures need the same care. Their meaning depends on the number of retained directions, whether the object is an activation or a concept vector, and whether a random-control contribution has been subtracted. A result from one decomposition is not the fraction of the model's information that matters.

There are more immediate limits. A complex concept may lack a single-token name. Several words appearing together do not tell us how their meanings are bound. Even a readout containing words for a spider, legs, and eight leaves the relationship to be tested. Conversely, failing to read a word does not establish that related information is absent.

For product teams, distinguishing these forms of evidence is useful. Behavior tests establish what a system accomplishes. Reasoning text provides process clues to inspect. Internal tools help locate and intervene on candidate mechanisms. The evidence should constrain itself: a plausible readout with no corresponding behavioral effect calls for a revised explanation. A harmful behavior cannot be declared safe because a lens failed to surface a suspicious word.

J-Lens makes a broad question more testable. We can ask where a direction appears, how downstream computation uses it, and what changes when it is edited. The word in the face example is a clue worth following.

The source draft was dated September 20, 2026. This article was revised and its diagrams prepared on November 5 using the paper and implementation available then. The display date does not establish historical public availability.

Further reading

View all