Demo: A Policy Creates Its Own Data

In supervised learning, a fixed dataset exists before the model. In reinforcement learning, a policy changes which states, actions, and rewards are observed.

We run the same CartPole environment and starting seeds under a state-independent policy and a simple feedback policy.

import gymnasium as gym
import numpy as np
import matplotlib.pyplot as plt

plt.style.use("../figure_style.mplstyle")
env = gym.make("CartPole-v1")

Two Action Rules

CartPole has two actions. The feedback rule chooses one from the pole angle and angular velocity. The random rule ignores the state.

def random_policy(state, rng):
    return int(rng.integers(2))

def feedback_policy(state, rng):
    angle, angular_velocity = state[2], state[3]
    return int(angle + angular_velocity > 0)

One Episode

Reset to a chosen starting seed, apply the policy until termination or the time limit, and add the rewards.

def run_episode(policy, seed):
    state, _ = env.reset(seed=seed)
    rng = np.random.default_rng(seed + 10_000)
    total_return = 0.0
    done = False
    while not done:
        action = policy(state, rng)
        state, reward, terminated, truncated, _ = env.step(action)
        total_return += reward
        done = terminated or truncated
    return total_return

Question: How Does a Policy Change the Experience It Collects?

The feedback policy uses pole angle and angular velocity, but it is still evaluated under varying initial states and dynamics. Compare the return distributions and ask which policy reaches long-survival trajectories more often.

seeds = range(100)
policies = {"State-independent policy": random_policy, "Feedback policy": feedback_policy}
returns = {}
for name, policy in policies.items():
    episode_returns = []
    for seed in seeds:
        episode_returns.append(run_episode(policy, seed))
    returns[name] = np.array(episode_returns)
env.close()

The two policies use the same 100 starting seeds. Plot the total rewards collected in their episodes.

fig, ax = plt.subplots(figsize=(7, 3.2))
bins = np.arange(0, 526, 25)
ax.hist(returns["State-independent policy"], bins=bins, alpha=0.65,
        label="State-independent policy", hatch="//")
ax.hist(returns["Feedback policy"], bins=bins, alpha=0.65,
        label="Feedback policy")
ax.set_xlabel("Episode Return (Steps Survived)")
ax.set_ylabel("Episodes")
ax.legend(frameon=False)
plt.show()

for name, values in returns.items():
    print(f"{name}: mean {values.mean():.1f}, median {np.median(values):.1f}, "
          f"range {values.min():.0f}--{values.max():.0f}")

State-independent policy: mean 22.4, median 17.0, range 9--58
Feedback policy: mean 493.1, median 500.0, range 320--500

Answer. Even a deterministic feedback rule produces a distribution of returns because episodes begin and evolve differently. Its state-dependent actions shift that distribution toward trajectories the state-independent policy rarely visits.

A policy is therefore both a decision rule and a data-collection rule. It cannot learn consequences of states and actions that its own behavior almost never reaches.