\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[a4paper, total={6in, 8in}]{geometry}
\usepackage{amsmath, amsfonts}
\usepackage{hyperref, graphicx}

\title{CSCI 145 Problem 21: Audit a Scaling Memo}
\author{} % TODO: Put your name here
\date{\today}

\begin{document}
\maketitle

\subsection*{Submission Instructions}
Please upload your work to Gradescope by \textbf{11:59pm on DATE}. % TODO: date
\begin{itemize}
\item Turn in a single PDF of \textit{handwritten} solutions. Typed (including \LaTeX) solutions will not be accepted.
\item Problems are grouped into three-week \textbf{cycles}. Once per cycle, you will meet with the grader and present the solution to \emph{one} problem from that cycle, selected at random when you arrive; bring your work and be ready to explain any of them.
\end{itemize}

\newpage \section*{Problem 21: Audit a Scaling Memo \hfill \textnormal{2 points}}

A lab plans a transformer run using the empirical loss model
$$
\mathcal L(N,D)=\mathcal L_\infty+aN^{-\alpha}+bD^{-\beta},
$$
where $N$ is the non-embedding parameter count and $D$ is the number of training tokens. Its memo assumes $C=6ND$ and concludes that the compute-optimal choices scale as
$$
N\propto C^{\beta/(\alpha+\beta)},
\qquad
D\propto C^{\alpha/(\alpha+\beta)}.
$$
It then says the recommendation is unchanged if context length grows from $2{,}048$ to $131{,}072$ tokens.

The allocation is correct under the memo's compute model, including constants, when
$$
N^*=\left(\frac{\alpha a}{\beta b}\right)^{1/(\alpha+\beta)}
\left(\frac C6\right)^{\beta/(\alpha+\beta)},
\qquad
D^*=\left(\frac{\beta b}{\alpha a}\right)^{1/(\alpha+\beta)}
\left(\frac C6\right)^{\alpha/(\alpha+\beta)}.
$$
Audit the long-context conclusion rather than re-deriving this optimization. Rebuild the training-compute constraint from a transformer block of width $d$, depth $L$, and sequence length $n$, accounting separately for parameter projections and attention mixing. Use $N=\Theta(Ld^2)$ to identify the regime in which $C=6ND$ is a defensible approximation and the regime in which the omitted term can change the allocation.

End with a recommendation about what the lab must remeasure or re-optimize before accepting the long-context plan. A constants-free asymptotic audit is sufficient.

\paragraph{Results you may use.}
You may use the per-layer costs $O(nd^2)$ for projections and $O(n^2d)$ for attention mixing.

%\input{solutions/solution21}
\end{document}
