\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[a4paper, total={6in, 8in}]{geometry}
\usepackage{amsmath, amsfonts}
\usepackage{hyperref, graphicx}

\title{CSCI 145 Problem 14: Filter the Interpolation Peak}
\author{} % TODO: Put your name here
\date{\today}

\begin{document}
\maketitle

\subsection*{Submission Instructions}
Please upload your work to Gradescope by \textbf{11:59pm on DATE}. % TODO: date
\begin{itemize}
\item Turn in a single PDF of \textit{handwritten} solutions. Typed (including \LaTeX) solutions will not be accepted.
\item Problems are grouped into three-week \textbf{cycles}. Once per cycle, you will meet with the grader and present the solution to \emph{one} problem from that cycle, selected at random when you arrive; bring your work and be ready to explain any of them.
\end{itemize}

\newpage \section*{Problem 14: Filter the Interpolation Peak \hfill \textnormal{2 points}}

Suppose $\mathbf{y}=\mathbf{X}\mathbf{w}^\star+\boldsymbol{\eta}$, where $\mathbf{X}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^\top$ and $\boldsymbol{\eta}$ has mean zero and covariance $\sigma_\eta^2 \mathbf{I}$. Instead of the minimum-norm interpolator from class, consider ridge regression:
$$
\hat{\mathbf{w}}_\lambda=\arg\min_\mathbf{w}\bigl\{\|\mathbf{X}\mathbf{w}-\mathbf{y}\|_2^2+\lambda\|\mathbf{w}\|_2^2\bigr\},
\qquad \lambda>0.
$$

Audit the claim ``any positive ridge penalty removes the interpolation peak at no statistical cost.'' Express $\hat{\mathbf{w}}_\lambda$ in the singular-vector coordinates of $\mathbf{X}$ and use those coordinates to find the bias, variance, and mean-squared coefficient error in a direction with singular value $s_i>0$ and signal coefficient $a_i=\mathbf{v}_i^\top \mathbf{w}^\star$. Determine that direction's best penalty, including the case $a_i=0$, and use the result to explain why one penalty generally cannot be best for every signal direction.

Give a precise verdict about the claim. Your explanation should identify both the unstable behavior that ridge suppresses near interpolation and the signal it can sacrifice, including what happens to signal in the null space of $\mathbf{X}$.

\paragraph{Results you may use.}
You may quote the normal equations and the SVD identities. If $z_i=\mathbf{u}_i^\top\boldsymbol{\eta}$ is the noise projected onto a left singular vector, then
$$
\mathbb E[z_i]=0,
\qquad
\mathbb E[z_i z_j]=\sigma_\eta^2\mathbb{1}\{i=j\}.
$$

%\input{solutions/solution14}
\end{document}
