Gemma Scope modded

An introspection harness for language models

  1. 1WatchWhat forms inside Gemma at each layer.
  2. 2Self-readingGemma gets its own state as numbers and is asked what it holds.

Watch

Gemma reads the question

Each word moves through Gemma’s layers one at a time. Play one of the recorded questions.

The words Gemma leans toward at each layer come from a lens in the logit-lens and tuned-lens line, after Anthropic’s Jacobian lens (Gurnee et al., 2026). The features are Google DeepMind’s Gemma Scope 2, and the trace of attention and units is an activation trace like TransformerLens-style tools.

See the detail
—
attention arcs · first-token share dimmed · most heads rest on the start token

Trace

Left: attention; center: total; right: units.

Features

Layer strength bar; IDs list top words.

Words

Three strongest Lens guesses per layer.

Prompt
Gemma’s reply

Underneath

Each state is a mix of Gemma Scope 2 features. Pick a word and a layer to see their shares and the part left unexplained.

Coming tonight: feature-share data is not available yet.

See the detail

The mix, by feature

Feature IDs, their share of this state, and the words linked to each ID.

What it read vs what it worked out

That the days of the week form a ring inside language models is a known finding (Engels et al., 2024). We test it here on Gemma 3 27B: on this model the ring shows only in the early layers. Later, the Lens favors the computed day before the answer, while the input token keeps its original day.

See the detail

Can it read itself?

Handing a model one of its own hidden states and asking what it holds builds on earlier work: Patchscopes (Ghandeharioun et al., 2024), SelfIE (Chen et al., 2024) and studies of introspection (Lindsey, 2025; Macar et al., 2026; Singh, Linzen and Ravfogel, 2026). Here it is tried on Gemma with simple tasks whose true value is known at every token.

Coming tonight: self-reading examples are not available yet.

See the detail

Coming tonight: answer-zone layers, matched pairs and causal swaps.

Coming tonight: refusal comparisons and forced-choice day scores.

See the detail

Self-reading detail will appear here when its data is available.

Can it change itself?

4B session chose PASS when the feature menu was unreadable.Read-back did not confirm a changed token.

Letting a model choose edits to its own features has been explored before (Black and Bloom, 2026), and steering with sparse-autoencoder features is established work (for example CorrSteer, Cho et al., 2026). Here it is tried in one recorded 4B session: a feature menu lets Gemma set a feature, undo it, or pass, followed by a read-back check. The menu was unreadable, Gemma replied PASS, and no token change was confirmed.

See the detail