# Benchmark PCA figure

The source is the benchmark table released with Ruan, Maddison, and Hashimoto's *Observational Scaling Laws and the Predictability of Language Model Performance* (NeurIPS 2024).

- Paper: https://arxiv.org/abs/2405.10938
- Repository: https://github.com/ryoungj/ObsScaling
- Pinned revision: `4d6e1e43fd2635d04654aa77d1df9d5266ea0382`
- Original file: https://github.com/ryoungj/ObsScaling/blob/4d6e1e43fd2635d04654aa77d1df9d5266ea0382/eval_results/base_llm_benchmark_eval.csv
- Local, unmodified copy: [benchmark_scores.csv](benchmark_scores.csv)
- Repository license: Apache 2.0, included as [LICENSE-ObsScaling](LICENSE-ObsScaling).

The selected columns are MMLU, ARC-C, HellaSwag, Winograd, TruthfulQA, GSM8K, XWinograd, and HumanEval, in that order.
“Winograd” is the source table's name for WinoGrande.
The pinned release contains 148 model rows, of which 124 have finite scores on all eight benchmarks.
The calculation uses those 124 complete rows, with no imputation.
This is an illustration using the released data, not a reproduction of the paper's original 77-model experiment.

Scores are converted from fractions to percentages.
Each column is centered by subtracting its mean across the 124 models, without scaling by its standard deviation.
The first three right singular vectors of this centered matrix form V_3.
The reconstruction is X V_3 V_3^T, with column means added back for display.
These are reconstructions of the data used to fit PCA, not predictions on held-out models or unobserved tests.
The tables display scores in percent, rounded to one decimal place; computed reconstructions are not clipped.

The three components retain 97.3556% of the squared variation of the centered matrix.
The root mean squared reconstruction error across all 124 × 8 entries is 2.667 percentage points.
The tables display three illustrative model rows: Llama 2 7B (`meta-llama/Llama-2-7b-hf`), Llama 3.1 70B (`meta-llama/Meta-Llama-3.1-70B`), and CodeLlama 34B (`codellama/CodeLlama-34b-hf`).
They show the MMLU, ARC-C, and HumanEval columns.
Horizontal, vertical, and diagonal ellipses indicate the omitted benchmark columns and model rows.
All 124 complete rows contribute to fitting the directions and calculating these statistics.
The displayed models' measured and reconstructed scores on all eight benchmarks, column means, principal directions, full list of fitted models, and the source CSV's SHA-256 hash are recorded in [benchmark_pca.json](benchmark_pca.json).

To regenerate both the figure and the numerical summary from the course repository, run:

```bash
tools/run_compute.sh benchmark-pca bash -c 'cd topics/15_low_rank_approximation && python3 -c "from generate_images import benchmark_reconstruction; benchmark_reconstruction()"'
```

The generator checks that the directions are orthonormal, the squared residual equals the sum of the discarded squared singular values, and retaining all eight directions reconstructs every measured score.