Composition lab

Learning to turn

Holonomy attention gives every token a rotation and reads the products of those rotations in the order the tokens appear. Carry a frame along a sequence to see what it computes, then compare the sixty rotations one head learned with the true symmetries of an icosahedron.

Model

Carry a frame

Each token is one of the sixty rotations. To relate a group of tokens, the layer multiplies their rotations in sequence order. That is the same as walking along the sequence with a frame and turning it only at the chosen tokens: the frame at the end is the product.

Turning at tokens 1, 4 and 6, the frame comes back unturned. These three moves undo each other.

The same moves with the first two swapped leave it turned by 120°: the term depends on the order.

marks each token that belongs to three tokens, in order, whose moves undo each other: what the task asks the layer to find. Click tokens to choose them.

The computation

The layer never lists the groups. A running product from the left collects, for every size, the ordered products before each token; one from the right collects those after it; and a group containing token i is one of each with M_i between them. Every size up to K is read out into the token's features. Paper, section on holonomy attention

X

M

scan \to

L_{a}

\leftarrow scan

R_{b}

L_{a} M_{i} R_{b}

F_{r}

Y

W_{A}, w_{g}

M_{i}

α_{r}, W_{o}

M_{t} = g_{t} e^{A_{t}}

L_{a} (t) = L_{a} (t - 1) + L_{a - 1} (t - 1) M_{t}

R_{b} (t) = R_{b} (t + 1) + M_{t} R_{b - 1} (t + 1)

F_{r} (i) = \sum_{a + b = r - 1} L_{a} (i) M_{i} R_{b} (i)

Y_{i} = \sum_{r \leq K} α_{r} ⊙ W_{o} vec F_{r} (i)

The sixty rotations, ideal and learned

Ideal symmetryComputed from the group law

Ideal reference Switch to Recorded to compare

The solid dot tracks one vertex. The open dot marks its starting point. A symmetry returns the shape to itself while moving its vertices.

Choose a rotationAll 60 group elements

Same moves. Different ending.

Apply rotation A then B, or B then A. The marked vertex ends in different places: these moves do not commute. A is element 01; B is element 12.

A → BAfter both moves

B → AAfter both moves