Demo: Learned Decision Regions on a Spiral

Two interleaved spirals cannot be separated by a straight line. We will use the example to see what a hidden ReLU layer adds, how its decision regions develop, and why width matters.

Question 1: Why Is This Data Difficult for a Linear Classifier?

Inspect the two classes before fitting. Any single line that separates one part of a spiral must cut through another part.

import numpy as np
import matplotlib.pyplot as plt

plt.style.use("../figure_style.mplstyle")
rng = np.random.default_rng(11)

n_per_class, noise = 150, 0.15
X, y = [], []
for c in [0, 1]:
    r = np.linspace(0.15, 1.0, n_per_class)
    t = np.linspace(c * np.pi, c * np.pi + 3.0 * np.pi, n_per_class) + rng.normal(0, noise, n_per_class)
    X.append(np.column_stack([r * np.sin(t), r * np.cos(t)]))
    y.append(np.full(n_per_class, c))
X, y = np.vstack(X), np.concatenate(y)

The two classes wind around each other. Predict where a straight boundary would fail.

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.scatter(X[y == 0, 0], X[y == 0, 1], c=y[y == 0], cmap="RdBu", vmin=0, vmax=1, s=14, label="Class 0")
ax.scatter(X[y == 1, 0], X[y == 1, 1], c=y[y == 1], cmap="RdBu", vmin=0, vmax=1, s=14, marker="^", label="Class 1")
ax.set_xlabel("$x_1$")
ax.set_ylabel("$x_2$")
ax.legend(frameon=False)
plt.show()

Answer. The classes alternate as they wind around the origin, so no half-plane contains one class without also containing points from the other.

Question 2: How Far Does a Linear Boundary Get?

We first train Topic 10’s softmax classifier. Its scores are linear functions of the two coordinates, so its class boundary must be one line.

def softmax(Z):
    Z = Z - Z.max(axis=1, keepdims=True)
    E = np.exp(Z)
    return E / E.sum(axis=1, keepdims=True)

Y = np.zeros((len(y), 2))          # one-hot labels
Y[np.arange(len(y)), y] = 1

rng_linear = np.random.default_rng(1)
W = rng_linear.normal(0, 0.1, size=(2, 2))
b = np.zeros(2)
for epoch in range(4000):
    P = softmax(X @ W + b)
    dZ = (P - Y) / len(X)
    W -= 1.0 * (X.T @ dZ)
    b -= 1.0 * dZ.sum(0)

linear_accuracy = np.mean(softmax(X @ W + b).argmax(axis=1) == y)
print(f"linear model, training accuracy: {linear_accuracy:.1%}")

linear model, training accuracy: 66.3%

Answer. The linear model performs only modestly better than chance because its boundary cannot follow the repeated turns.

Question 3: What Does a Hidden Layer Add?

A ReLU layer creates many piecewise-linear features before the final softmax. The code implements the same forward and backward equations from the reading and saves a few training snapshots.

def train(X, Y, hidden_dim, n_epochs=4000, alpha=1.0, seed=1):
    n, d = X.shape
    k = Y.shape[1]
    rng2 = np.random.default_rng(seed)
    W1 = rng2.normal(0, np.sqrt(2 / d), size=(d, hidden_dim))
    b1 = np.zeros(hidden_dim)
    W2 = rng2.normal(0, np.sqrt(2 / hidden_dim), size=(hidden_dim, k))
    b2 = np.zeros(k)
    snapshots = {}
    for epoch in range(n_epochs + 1):
        Z1 = X @ W1 + b1
        H = np.maximum(0, Z1)
        P = softmax(H @ W2 + b2)
        if epoch in (0, n_epochs // 20, n_epochs // 4, n_epochs):
            snapshots[epoch] = (W1.copy(), b1.copy(), W2.copy(), b2.copy())
        # The final snapshot already contains n_epochs updates.
        if epoch == n_epochs:
            break
        dZ2 = (P - Y) / n
        dZ1 = (dZ2 @ W2.T) * (Z1 > 0)
        W2 -= alpha * (H.T @ dZ2)
        b2 -= alpha * dZ2.sum(0)
        W1 -= alpha * (X.T @ dZ1)
        b1 -= alpha * dZ1.sum(0)
    return snapshots

def predict(params, points):
    W1, b1, W2, b2 = params
    H = np.maximum(0, points @ W1 + b1)
    return softmax(H @ W2 + b2)[:, 1]

snapshots = train(X, Y, hidden_dim=64)
trained = snapshots[4000]
print("snapshots at epochs:", list(snapshots))
print(f"64 hidden neurons, training accuracy: {np.mean(predict(trained, X).round() == y):.1%}")

snapshots at epochs: [0, 200, 1000, 4000]
64 hidden neurons, training accuracy: 100.0%

Answer. With enough hidden units, the network can assemble several straight-edged regions into a nonlinear boundary and fit the spiral far better than the linear model.

Question 4: How Does the Boundary Develop During Training?

The backgrounds below show the predicted probability of class \(1\) at four epochs. Compare the initially arbitrary regions with the final shape.

xx, yy = np.meshgrid(np.linspace(-1.3, 1.3, 200), np.linspace(-1.3, 1.3, 200))
grid = np.column_stack([xx.ravel(), yy.ravel()])

fig, axes = plt.subplots(1, 4, figsize=(9, 2.6))
for ax, epoch in zip(axes, snapshots):
    probs = predict(snapshots[epoch], grid).reshape(xx.shape)
    shown = ax.imshow(probs, extent=[-1.3, 1.3, -1.3, 1.3], origin="lower", cmap="RdBu", vmin=0, vmax=1)
    ax.scatter(X[y == 0, 0], X[y == 0, 1], c=y[y == 0], cmap="RdBu", vmin=0, vmax=1, s=5)
    ax.scatter(X[y == 1, 0], X[y == 1, 1], c=y[y == 1], cmap="RdBu", vmin=0, vmax=1, s=5, marker="^")
    ax.set_xlabel(f"Epoch {epoch}")
    ax.set_xticks([])
    ax.set_yticks([])
fig.colorbar(shown, ax=axes, fraction=0.025, pad=0.02, label="Predicted probability of class 1")
plt.show()

Answer. Training rearranges the piecewise-linear regions until they trace the two arms of the spiral.

Question 5: How Much Hidden Width Is Needed?

Reduce hidden_dim and repeat the fit. A narrow layer supplies fewer ReLU boundaries from which the classifier can build its decision region.

hidden_dim = 4    # change me!

narrow = train(X, Y, hidden_dim=hidden_dim)
params = narrow[max(narrow)]
probs = predict(params, grid).reshape(xx.shape)

fig, ax = plt.subplots(figsize=(7, 4.3))
shown = ax.imshow(probs, extent=[-1.3, 1.3, -1.3, 1.3], origin="lower", cmap="RdBu", vmin=0, vmax=1)
ax.scatter(X[y == 0, 0], X[y == 0, 1], c=y[y == 0], cmap="RdBu", vmin=0, vmax=1, s=10)
ax.scatter(X[y == 1, 0], X[y == 1, 1], c=y[y == 1], cmap="RdBu", vmin=0, vmax=1, s=10, marker="^")
ax.set_xlabel(f"{hidden_dim} hidden neurons, epoch {max(narrow)}")
ax.set_xticks([])
ax.set_yticks([])
fig.colorbar(shown, ax=ax, fraction=0.04, pad=0.02, label="Predicted probability of class 1")
plt.show()

print(f"{hidden_dim} hidden neurons, training accuracy: {np.mean(predict(params, X).round() == y):.1%}")

4 hidden neurons, training accuracy: 66.0%

Answer. Four hidden neurons provide too few pieces to follow the spiral, so this network performs little better than the linear model. Width controls the set of decision regions the network can represent. Optimization then chooses among them.