Recently we released an interim progress post on the chain lens, which is a new interpretability tool I've been working on for the last month. The chain lens takes existing token lenses like the J-lens and augments them with a multi-token prediction head, turning them into multi-token phrase lenses.

At its core, this is an incredibly simple idea: we have this tool that can turn a hidden state into a single-token readout (the J-lens), we'd like it to be multi-token, and there's a component in the model that literally has "multi-token" in the name, can we use that... and it turns out yes, we can.

For more technical details you can read the post, but because of it I've been thinking more about interpretability and our understanding of models. Why did we find this? If it's such a simple idea, why did it take until now? Why does such a simple idea even work?

Much of the belief that AI will be inherently hard to understand seems to come from 'black box trauma': the death of symbolic AI, "every time I fire a linguist the performance of the system goes up," the bitter lesson, the failure of eXplainable AI, the big blob of compute and scaling hypotheses. Early setbacks in model interpretability, like the consensus that OpenAI's sentiment neuron was rare and most neurons are in superposition, made the situation seem hopeless. Why should we expect to be able to understand trillion-parameter models? It's not like humans would have any hope of understanding random trillion-line Python programs. The words must be riding on top of an inscrutable ocean.

But recently, there's been a flood of new avenues for understanding models that were hiding in plain sight. J-lens found a simple improvement to the logit lens that no one had noticed for six years. Scalable interp NLAs and oracles are independently starting to work really well. Models are getting good at introspection and self-explaining their mechanisms. SAEs and transcoders are still kicking around, but neuron interpretability also turns out to work surprisingly well. Something else is going on.

Why are there so many different ways to interpret transformers? While each technique is different, there's a guiding principle uniting them: transformers are built around a very nice object. The d_model-width semantic vector is the Lego brick of interpretability, made out of words.

Theoretically these vectors could contain anything and be completely incompatible with each other, but in practice they seem to be substantially made of words. Say you average together a bunch of these vectors from different positions of models writing about lighthouses. The resulting vector isn't the random junk you'd get if you tried to average together books or .exe files about lighthouses, but instead a useful, purified "lighthouse" concept vector. This vector can be used to steer the model to think about lighthouses. It will light up a probe when the model is thinking about lighthouses. It can be read out by a token lens - because the lenses, built on the model's lm_head machinery, also take d_model vectors. Want more detail? You can plug the vector into the oracle lens, or Patchscope it into the model to self-explain, because those techniques - developed on the residual stream - again take d_model vectors. Find a lighthouse neuron or attention head, and the write or query vector will be fairly similar to the lighthouse vector you made by hand - because neuron writes and attention head query vectors, among others, are also d_model sized and have similar semantics, so you can even use the vector you made to search for neurons and heads in the model. SAE features? d_model vectors, so they can be explained by many of these techniques. Want to dig up an old technique like ICA? It works, and the components... you'll never believe this... d_model vectors that work with all these techniques.

I don't want to paint too simple a picture of interpretability. It's hard, experiments fail, things are confusing, internal geometry is sometimes complex. We still don't yet rigorously understand many things about LLMs that are prominent in our theories. (What is a "persona" being selected or a "User" to be corrigible to?) It seems necessary for there to be objects in LLMs that are sub-linguistic, like representations of entropy, if only for linguistic bookkeeping. We could also be missing things due to streetlight effects, linguistic objects being easier to find.

But many things that could have been sub- or non-linguistic actually seem to bind to language machinery, like directions in spatial representations, vision patches in VLMs, or even beat saber concepts. LLMs seem to be substantially made of language in a deep, robust, generalizing way. There has to be some point where linguistic methods show enough results and causal efficacy that the burden of proof shifts to the camp pulling an alien-maximizer-of-the-gaps. If the nonlinguistic substructure is so important, at what point will we see evidence of it? If the talking is just a thin shell, why does intervening based on it work so well?

Lego's guiding principle is that all elements fit together, can be used in multiple ways, and can be built together. LLM interpretability increasingly feels like this to me: playing in a Lego set made of word vectors, where everything snaps together through the interface of language.