Cubic attention

One head scores every query against every ordered pair of other cards. Pick a card, a voxel or a membrane; all views follow.

Attention scores

A

Drag to orbit. Click a bright voxel to select it.

Linked view The selected hyperedge is the selected voxel.

Selected interaction

Attention score

A_{ij k} = ⟨ Q_{i}, K_{j}^{1} ⊙ K_{k}^{2} ⟩

A query scores a pair of keys.

Value interaction

D_{j k} = V_{j}^{1} ⊙ V_{k}^{2}

Pairs of values interact before aggregation.

Value features

V_{j}^{1}

V_{k}^{2}

Resulting feature vector

D_{j k}

Selected quantities

$A_{ij k}$	Scalar score	4.00
$P_{ij k}$	Scalar weight	23.9%
$P_{ik j}$	Swapped roles	23.9%
Sink	No pair fits	2.5%
SET	Do i, j, k form one?	Yes

Constructed head

Built by hand, not learned: 16 trilinear channels compute $\sum_{t} cos \frac{2 π}{3} (a_{t} + b_{t} + c_{t})$ , which is 4 for every SET and at most 2.5 otherwise. Switch the head to Trained to see what gradient descent finds.

Query slices of A

Query slices of $A$