We opened a hybrid vision-language model to find out where an image lives while the model talks about other things. The answer surprised us: the model's huge recurrent memory only looks like one. It retains almost nothing about the image beyond a few tokens; what looks like memory is the attention cache re-supplying the picture at every step.
Show a model a photograph. Let the conversation drift for a paragraph or two. Then ask what was in the picture.
For a transformer this question is easy, because a transformer never lets go of anything: every token it has read, including every patch of the image, sits in its attention cache as a key-value entry that any later step can look at. The picture is not remembered so much as still there.
Hybrid models make a different bargain. Zamba2-VL-7B, the model this article opens up, has 81 decoder layers, and every one of them carries a Mamba2 mixer, a recurrent layer (explained from scratch in the appendix) that compresses whatever it reads into a fixed block of 458,752 numbers. Each new token partially overwrites that block. Only 13 of the 81 layers also carry attention. This design is why such models promise constant-memory inference, and it raises a basic question: while the conversation drifts, where does the picture live?
We spent under $30 of rented GPU time finding out, and this article walks through the answer in the order we found it. First, the model's own weights tell us how fast its recurrent memory must fade. Second, when we actually measure that memory, it holds far more of the picture than the weights allow, which should be impossible. Third, one deletion experiment resolves the contradiction, and settles where the picture really lives.
One diagram explains everything that follows, so it is worth thirty seconds. As tokens stream in, the attention layers append one key-value entry per token and keep all of them. The recurrent layers hold a state of fixed size: the same 458,752 numbers per layer at every position, rewritten at every token. Watch what happens to the image in each channel.
This measurement is easy to get wrong, so the design of the experiment matters. Our basic trial has three parts in a row: a COCO photograph, which becomes roughly 400 visual tokens; then a stretch of neutral filler text whose length d we control; then an eight-way question about the picture, such as which object it showed.
Just before the question is processed, we read the recurrent state with a linear probe. A linear probe is the simplest possible reader: a classifier that computes a weighted sum of the numbers in the state and predicts the answer from that sum alone, with no other computation. We train it on some trials and test it on held-out ones. If even this simple reader recovers the answer, the answer is present in the state in an easily readable form. The widget below shows the idea; drag the slider to see what distance does to the reading.
Two design choices prevent false positives. The eight candidate answers are fixed within a block of trials and each is correct exactly once, so the chance rate is exactly 12.5% and a probe cannot beat it by learning the dataset's habits. And every experiment carries a control probe placed before the image: if the instrument is clean, that probe must read exactly chance. It does, at every model scale we tested.
Start at distance zero, the instant the image has just been read. We ask the state three kinds of question, always with the same method: an eight-way multiple choice with its own candidate list, read by its own linear probe trained exactly as above. Which object is in the picture, with eight object names as candidates. How many of a named object, with eight counts as candidates. And where a named object sits, with eight regions of the frame as candidates.
The state answers only the first one well. A probe across all 81 layers reads the object's identity at 83.1% against the 12.5% chance rate, and the best single layer, layer 64, reads 92.5% on its own. The probes for count and position, trained the same way on the same trials, barely beat chance.
Now let text flow past the image. In our trials this is the filler: the d tokens of neutral prose between the image and the question. Every one of those tokens partially overwrites the recurrent state, so the gist must fade. How fast? Here the architecture does something unusual: it tells you in advance. The Mamba2 recurrence updates each head's state as
S ← S · eΔA + write
The multiply is the only operation that ever removes information, and both of its ingredients are inspectable: A is a stored weight, and the gate Δ can be measured on ordinary text. So each of the model's 9,072 state heads has a half-life, ln 2 / (|A|·Δ̄), in tokens, computed from the checkpoint with nothing fitted to any decay curve.
So the weights predict a memory measured in single-digit tokens. And when we probe the state at increasing distance, the reading does sink, at every model scale, in roughly the predicted shape. But look at where the curves settle.
Sixty-four tokens is around ten half-lives for a typical head, and ten half-lives leaves a thousandth of the original trace. So at that distance the probe should find essentially nothing. Instead it identifies the object a quarter of the time. Either the arithmetic is wrong, or something is refilling the state.
Here is a test that can tell those two possibilities apart. Every one of the model's 9,072 heads has its own predicted half-life, so we can sort them. We call a head fast if its half-life is short, around two tokens: its state should empty almost immediately. We call a head slow if its half-life is long, around sixteen tokens: it should hold information the longest. Then we probe the fast group and the slow group separately. If the state genuinely retains the image, the fast heads should lose it first, and whatever signal survives at long distance should be carried by the slow heads.
Before running this test, we have to deal with a complication, because it fooled us at first. Recall that a head's half-life depends on two numbers: the decay weight A and the gate Δ. The gate does double duty. It sets how fast the head forgets, and it also sets how strongly the head writes into its state. A head with a large gate therefore forgets faster and writes harder, by about a factor of seven across the model. Why does writing harder matter? Because a head that writes with larger numbers leaves a stronger signal in its state, and a stronger signal is easier for a probe to read, regardless of where the information came from or how old it is. So sorting heads by half-life also, accidentally, sorts them by signal strength. When we first did the naive sort, the fast heads read better than the slow ones, the opposite of the decay prediction. That reversal had nothing to do with memory. It was the write strength.
The fix is to compare only heads that write equally hard. We keep the heads whose gate Δ falls in a narrow middle band, so their write strengths are nearly equal, and within that band we split by the decay weight A alone. This still gives a fast group and a slow group, which we call the fast and slow strata, and their predicted half-lives still differ sevenfold: 2.3 tokens against 16.4. But now the only meaningful difference between the two groups is how fast they forget.
This is the central finding of the work. The ordering of the strata follows the weights, so the decay calculation is measuring something real. The amount that survives is off by nine orders of magnitude, so what the probe reads at distance cannot be retention. Something else must be writing the image into the state, over and over, long after the image has passed.
Only one component of this model still holds the image at distance: the attention cache. So we ran the deletion that separates storing from relaying. Each trial runs twice, identically. In one run, at the exact instant the image ends and before a single filler token is processed, we delete the image's key-value entries from all 13 attention layers. The recurrent state is never touched, and at the moment of the cut it is identical in both runs. If the state stores the picture, the readout downstream should not care.
Falling below chance tells us something specific. The heads were not merely degraded by the deletion; they are now reading out whatever the filler text wrote instead of the image. So whatever identity signal a probe finds in those heads at distance was arriving from the attention channel, freshly, at every step.
There is a fair objection to the deletion experiment. Deleting entries does not only remove the image: it also makes every attention cache shorter, and it changes how attention distributes its focus over the entries that remain. Maybe the probe's reading collapsed because we damaged the machinery, not because we removed a memory.
Here is the experiment that settles it. Run the deletion twice, on the same trials. In one version, delete the image's entries, exactly as before. In the other version, delete the same number of entries from the same caches, but delete text entries instead of image entries. Both versions do identical mechanical damage: same number of entries gone, same shorter caches, same disturbance to attention. So if damage were the explanation, both versions should push the reading down.
That is not what happens. The two deletions move the reading in opposite directions. Deleting the image's entries pushes it down, below chance. Deleting text entries pushes it up, above the no-deletion baseline, most likely because the text was competing with the image for the model's limited bandwidth, and removing it leaves the image easier to relay. Damage cannot push a reading up. The animation below shows the two versions with the real numbers.
One experiment remains, the most direct one. Run the model on two different images, pause both runs at the same point, and physically exchange one memory channel between them. Then let the first run answer its question and see whose picture it describes.
Swap the entire recurrent memory, all 81 layers, all 37 million numbers of it, and nothing happens: the model answers from its own image in 100 trials out of 100, as if the transplant had never occurred. Swap attention caches instead and there is a cliff: at seven of the thirteen layers, the model switches to describing the other run's image in 94 trials of 100.
Why does swapping 37 million numbers change nothing? The relay result already answers this. A transplanted state decays away within a handful of tokens, and is then rewritten from the attention channel, which the swap never touched. The two experiments agree on one mechanism.
It is not a memory of the image. It is a working summary of the image's gist, held for a few tokens at a time and continuously refreshed from the attention channel, which is the only place the picture durably lives. The state looks like a memory to a probe only because the relay keeps restocking it. Delete attention's copy, and the appearance of memory disappears with it.
This finding matters for a common optimization. An image adds hundreds of key-value entries to the attention cache, which makes vision-language models expensive to serve. A large body of work therefore speeds them up by deleting most of those entries, a technique called pruning. The justification is that the entries appear redundant: on standard benchmarks, you can delete many of them and accuracy barely moves.
Our results explain why pruning looks safe, and when it stops being safe. On this model, keeping a quarter of the visual entries costs almost nothing, at every distance we tested. Prune harder than that, though, and the model has to fall back on its recurrent state to fill the gap. Near the image this works, because the state is still holding the picture's gist. But the state forgets within a few tokens, so the further the question sits from the image, the less there is to fall back on. This is why the cost of heavy pruning is not one number: it roughly triples between a question asked immediately after the image and one asked 512 tokens later.
The whole argument in eight and a half minutes, narrated over animations.
The premise that a fixed-size state must eventually lose information is not ours. Jelassi et al. prove transformers can copy exponentially longer strings than state-space models; related lines show state tracking and in-context retrieval are the specific gaps, and that most of the quality gap between efficient architectures and attention is recall. What those results compare is architectures from the outside. What was missing is what happens inside one deployed hybrid: when it forgets, what it keeps while forgetting, and which component actually fails.
This article leans on one piece of machinery, the Mamba2 recurrence inside a hybrid decoder stack, so here it is from the beginning. You do not need this section to follow the argument above, but it explains where the half-life formula comes from, and where the two memories physically live.
Zamba2-VL-7B is a tower of 81 decoder layers, and a token passes through them in order, bottom to top. Every one of the 81 layers carries a Mamba2 mixer with its own recurrent state. At 13 fixed depths (layers 6, 11, 17, 23, 29, 35, 41, 47, 53, 59, 65, 71 and 77) the layer additionally runs ordinary attention. This is the whole hybrid design: recurrence everywhere, attention at a thin sprinkling of depths. The two memories the article keeps comparing are exactly these two pieces of hardware. Watch what one token does to each of them.
Between tokens, a Mamba2 layer carries a single block of numbers called the state, written S. The state is the layer's only memory: whatever the layer wants to remember about everything it has read so far must be stored in those numbers. In Zamba2-VL-7B each layer's state is split into 112 independent heads, and each head keeps its own 64×64 matrix; that is 112 × 64 × 64 = 458,752 numbers per layer, and 9,072 heads across the 81 layers.
When the next token x arrives, the layer computes three of its four working quantities from the token itself, and reads the fourth out of the checkpoint:
Step one is forgetting: the whole state is multiplied by eΔA, a number between zero and one, which shrinks everything the head currently holds. Step two is writing: the new content Δ·x·B is added on top. Then the output is read with C, which changes nothing. The animation below shows one head doing this, token after token.
Now the key point for this article. Because the shrink factor eΔA is applied once per token, surviving k tokens means being multiplied by it k times. The fraction of a memory that remains decays exponentially, and it falls to one half after exactly ln 2 / (|A|·Δ̄) tokens. That is the half-life used throughout this article. Both numbers in the formula are knowable in advance: A is stored in the checkpoint, and the average gate Δ̄ can be measured on ordinary text. Try the sliders below to see how the two numbers set the forgetting speed.
All of the code is public: github.com/VizuaraAI/visual-memory contains every instrument, experiment, figure, and paper source, plus this website. Every number comes from a scripted experiment on public checkpoints and public data, and the instruments read each checkpoint's geometry from its own configuration, which is what let the same code run at three scales.
lab/ instruments: telemetry, dynamics, probes, splice analysis/ decoding and statistics figures/ every figure, each function returning what it drew results/ raw JSON and feature archives paper/ the paper sources
Model: Zamba2-VL-7B (Apache-2.0) with its 2.7B and 1.2B siblings. Trials from COCO val2017. Total compute under $30 of rented H100 time. Extended paper · conference version.