Last lecture’s transformer used a learned table of position vectors. Today’s fix is RoPE: rotate the query at position \(m\) by the angle \(m\theta\), rotate the key at position \(n\) by \(n\theta\), and let the in-class identity \(\mathbf{R}_m^\top\mathbf{R}_n = \mathbf{R}_{n-m}\) do the rest. For one repeated token, the attention score matrix is Toeplitz: constant along every diagonal.
We use a small head, \(d = 8\), so the frequency ladder collapses to clean powers of ten, \(\theta_j = 10000^{-2j/8} = 10^{-j}\).
import numpy as npimport matplotlib.pyplot as pltimport scienceplotsfrom matplotlib.colors import LinearSegmentedColormapplt.style.use(["science", "no-latex"])TEAL, CARDINAL, GRAY ="#009090", "#9c1b33", "#c9c9c9"SCORE_CMAP = LinearSegmentedColormap.from_list("course_div", [CARDINAL, "white", TEAL])def rotation(angle):return np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])def block_rotation(m, thetas):# The RoPE matrix Theta_m: one 2x2 rotation block per frequency. d =2*len(thetas) R = np.zeros((d, d))for j, theta inenumerate(thetas): R[2*j:2*j+2, 2*j:2*j+2] = rotation(m * theta)return Rd =8thetas =10000.0** (-2* np.arange(d //2) / d)print("Frequency ladder:", thetas)rng = np.random.default_rng(30)q = rng.standard_normal(d) # one query content vectork = rng.standard_normal(d) # one key content vectorprint(f"||q|| = {np.linalg.norm(q):.6f}")
Frequency ladder: [1. 0.1 0.01 0.001]
||q|| = 3.530342
Rotation never changes a query’s length
In class we proved \(\|\mathbf{R}_m\mathbf{q}\| = \|\mathbf{q}\|\) with a 2x2 computation. The block-diagonal \(\mathbf{\Theta}_m\) inherits the property block by block. We check it at positions beyond those used in the hand calculation.
for m in [0, 1, 7, 100, 5000]:print(f"||Theta_{m} q|| = {np.linalg.norm(block_rotation(m, thetas) @ q):.6f}")
Position \(5000\) has wound the fast hand around the circle nearly \(800\) times, while the norm is unchanged. Rotation changes which keys a query aligns with, not its magnitude.
The score matrix of a repeated token
We now compute the full score matrix. Plant the same content \(\mathbf{q}, \mathbf{k}\) at sixteen positions (so the scores isolate what position alone contributes), rotate each copy by its own \(\mathbf{\Theta}_m\), and score every query against every key.
n_pos =16rotated_q = np.stack([block_rotation(m, thetas) @ q for m inrange(n_pos)])rotated_k = np.stack([block_rotation(n, thetas) @ k for n inrange(n_pos)])S = rotated_q @ rotated_k.T / np.sqrt(d)vmax = np.abs(S).max()fig, ax = plt.subplots(figsize=(6, 4.4))im = ax.imshow(S, cmap=SCORE_CMAP, vmin=-vmax, vmax=vmax)ax.set_xticks(range(n_pos)); ax.set_yticks(range(n_pos))ax.set_xticklabels(range(n_pos), fontsize=7)ax.set_yticklabels(range(n_pos), fontsize=7)ax.tick_params(which="both", bottom=False, left=False, top=False, right=False)ax.set_xlabel("Key position $n$")ax.set_ylabel("Query position $m$")cbar = fig.colorbar(im, ax=ax, fraction=0.045, pad=0.03)cbar.set_label("Attention score")plt.show()
Stripes, parallel to the main diagonal. We verify the Toeplitz pattern by measuring how much the entries vary along each of the \(31\) diagonals. While we are here, we can also check the class identity directly, entry by entry, against \(\mathbf{q}^\top\mathbf{\Theta}_{n-m}\mathbf{k}/\sqrt{d}\).
diag_stds = [S.diagonal(offset).std() for offset inrange(-n_pos +1, n_pos)]print(f"Largest std along any diagonal: {max(diag_stds):.2e}")identity_err =max(abs(S[m, n] - q @ block_rotation(n - m, thetas) @ k / np.sqrt(d))for m inrange(n_pos) for n inrange(n_pos))print(f"Largest |S[m,n] - q.Theta_(n-m).k/sqrt(d)|: {identity_err:.2e}")
Largest std along any diagonal: 1.43e-16
Largest |S[m,n] - q.Theta_(n-m).k/sqrt(d)|: 4.44e-16
Zero, up to floating point. Every diagonal is constant, and every entry equals the class identity’s prediction \(\mathbf{q}^\top\mathbf{\Theta}_{n-m}\mathbf{k}/\sqrt{d}\): the score depends on \(n - m\) and on nothing else.
The hands of the clock
Each 2D slice of the embedding is a hand advancing \(\theta_j\) radians per position. We track the fastest and second-fastest hands across the sixteen positions.
fig, axes = plt.subplots(1, 2, figsize=(7, 3.6))for ax, theta, caption inzip(axes, [1.0, 0.1], ["Fast hand: $\\theta_0 = 1$", "Slower hand: $\\theta_1 = 0.1$"]): ax.add_patch(plt.Circle((0, 0), 1, fill=False, color=GRAY, linewidth=1)) fast = theta ==1.0for m inrange(n_pos): angle = m * theta near_collision = fast and m in (0, 6) color = CARDINAL if near_collision else TEAL ax.plot(np.cos(angle), np.sin(angle), "o", color=color, markersize=4)ifnot (fast or m %5==0):continue# the slow hand's positions crowd together; label every fifth r_label =1.24if (fast and m ==6) else1.16 ax.text(r_label * np.cos(angle), r_label * np.sin(angle), str(m), ha="center", va="center", fontsize=7, color=color if near_collision else"black") ax.set_xlim(-1.5, 1.5); ax.set_ylim(-1.5, 1.5) ax.set_aspect("equal"); ax.set_xticks([]); ax.set_yticks([]) ax.set_xlabel(caption)plt.show()
The fast hand laps the circle twice, and positions \(0\) and \(6\) (cardinal) land closer together than any two consecutive positions do. One hand repeats itself, which is the aliasing Problem 22 turns into a formula. The slower hand fans the same sixteen positions out in unambiguous order, but into a wedge of only \(1.5\) radians (only every fifth position is labelled, since they crowd together). Fast hands distinguish nearby positions, slow hands distinguish distant ones, and RoPE uses a whole ladder of frequencies.
Varying the content and frequency scale
The stripes came from position, not from attention in general. Set VARY_CONTENT = True to give every position its own random token. Or leave it False and play with BASE (the ladder’s \(10000\)) or N_POS (the sequence length), and watch the stripes stretch and shrink.
VARY_CONTENT =True# change me!BASE =10000.0# change me too: try 100, or 2N_POS =16# or make the sequence longerthetas_wi = BASE ** (-2* np.arange(d //2) / d)rng_wi = np.random.default_rng(31)S_wi = np.zeros((N_POS, N_POS))for m inrange(N_POS):for n inrange(N_POS): q_mn = rng_wi.standard_normal(d) if VARY_CONTENT else q k_mn = rng_wi.standard_normal(d) if VARY_CONTENT else k S_wi[m, n] = block_rotation(m, thetas_wi) @ q_mn @ (block_rotation(n, thetas_wi) @ k_mn) / np.sqrt(d)vmax = np.abs(S_wi).max()fig, ax = plt.subplots(figsize=(6, 4.4))im = ax.imshow(S_wi, cmap=SCORE_CMAP, vmin=-vmax, vmax=vmax)ax.set_xlabel("Key position $n$"); ax.set_ylabel("Query position $m$")ax.set_xticks([]); ax.set_yticks([])cbar = fig.colorbar(im, ax=ax, fraction=0.045, pad=0.03)cbar.set_label("Attention score")plt.show()diag_stds = [S_wi.diagonal(offset).std() for offset inrange(-N_POS +1, N_POS)]print(f"Largest std along any diagonal: {max(diag_stds):.2e}")
Largest std along any diagonal: 1.38e+00
With varying content the largest diagonal deviation jumps from \(10^{-16}\) to about \(1.4\): the stripes dissolve, because content differences now sit on top of the positional pattern. In a trained model both effects coexist, with content deciding what to attend to and RoPE modulating it by relative position only.
Each query and key is rotated by its absolute position, but their comparison depends only on the relative offset because \(\mathbf{R}_m^\top\mathbf{R}_n = \mathbf{R}_{n-m}\).
