\questionbox{How do semantic associations emerge during the training of attention-based language models on natural language data?}

\begin{definition}[Attention-Based Transformer]
Given an input matrix $X \in \mathbb{R}^{T \times |\mathcal V|}$ and parameters $\Theta$, define
\begin{equation}
F_{\Theta}(X)=h^{(L)}W_O,
\end{equation}
where
\begin{equation}
h^{(l)} = h^{(l-1)} +
S\!\left(\Mask\!\left(h^{(l-1)}W^{(l)}h^{(l-1)\top} + \DM(P^{(l)})\right)\right)
h^{(l-1)}V^{(l)},
\qquad
h^{(0)} = X.
\end{equation}
Here, $S(\cdot)$ denotes softmax, $\DM(\cdot)$ maps a vector to a relative-position bias matrix, and $\Mask(\cdot)$ applies the causal mask.
\end{definition}

\begin{theorem}[Informal]
Under sufficiently small initialization, after $s$ gradient-descent steps with learning rate $\eta$, the early-stage parameters satisfy
\begin{align}
\|W_O - s\eta \bar B\|_F &\le 3 s^2 \eta^2, \\
\left\|V^{(l)} - \binom{s}{2}\eta^2 \bar\Phi^\top \bar B^\top\right\|_F &\le 12 s^3 \eta^3, \\
\left\|W^{(l)} - \left(3\binom{s}{4}+2\binom{s}{3}\right)\eta^4 \bar Q\right\|_F &\le 13 s^5 \eta^5 T, \\
\left\|P^{(l)} - \left(3\binom{s}{4}+2\binom{s}{3}\right)\eta^4 \Delta\right\|_F &\le 13 s^5 \eta^5 T.
\end{align}
The first term captures bigram statistics, whereas the higher-order terms capture contextual and position-dependent structure.
\end{theorem}

\begin{lemma}[Softmax Jacobian Norm]
Let $\sigma:\mathbb{R}^n \to \mathbb{R}^n$ be the softmax map. Then
\begin{equation}
\|J_{\sigma}(z)\|_2 \le \frac{1}{2}
\end{equation}
for every $z \in \mathbb{R}^n$.
\end{lemma}

\begin{lemma}[First Gradient Step]
After one gradient step,
\begin{equation}
W_O^{(1)} = W_O^{(0)} - \eta \nabla_{W_O}\mathcal L = \eta \bar B.
\end{equation}
\end{lemma}

\begin{proof}
At initialization all higher-order interaction terms vanish, so the gradient with respect to $W_O$ reduces to $-\bar B$. Substituting this quantity into the gradient update yields the claim.
\end{proof}

\paragraph{Learning objective.}
We study the next-token prediction loss under a causal attention mask and track how its gradient reveals increasingly structured statistics over the course of training.

\noindent\textbf{(1) Bigram mapping $\bar B$.}
The matrix $\bar B$ summarizes adjacent token transitions and serves as the first feature learned by the output layer.

\noindent\textbf{\textit{Three basis functions and their composition characterize the model's early behavior.}}

To formalize this intuition, we decompose the training objective as
\begin{equation}
\mathcal E = \mathcal E_{\mathrm{signal}} + \lambda \mathcal E_{\mathrm{bias}} + \gamma \mathcal E_{\mathrm{noise}}.
\end{equation}
Here, $\mathcal E_{\mathrm{signal}}$ measures predictive structure, $\mathcal E_{\mathrm{bias}}$ captures systematic positional effects, and $\mathcal E_{\mathrm{noise}}$ collects residual variability.

The decomposition separates token-level statistics from positional effects, which suggests that the first nontrivial feature should appear in the output projection.

\begin{equation}
\Psi = \Psi_{\mathrm{token}} + \Psi_{\mathrm{position}}.
\end{equation}

This observation leads to the following theorem.

\begin{theorem}
Suppose the initialization scale is at most $\sigma_0$ and the learning rate is $\eta \le \eta_0$. Then the output projection recovers $\Psi_{\mathrm{token}}$ before any position-dependent parameter grows beyond order $\eta^2$.
\end{theorem}

gap → object → formula → interpretation → theorem / claim → validation

To formalize this intuition, we define ...
We now express this quantity as ...
To understand how the error propagates, we analyze ...
The following decomposition separates ... into ...

Motivated by the discrepancy observed above, we express the error term as
\begin{equation}
E_t = \alpha\,\mathrm{diag}(\delta_t)\,K_t.
\end{equation}

Here, ... denotes ..., while ... represents ...
The first term captures ..., whereas the second term accounts for ...
where ||·||_F is ..., and B corresponds to ...

Here, $\Psi$ denotes the context-summary statistic, while $B$ represents the local next-token mapping.
The product $\Psi^\top B^\top$ therefore combines long-range context aggregation with local predictive structure.

Intuitively, this term increases when ...
In essence, Eq. (...) shows that ...
This expression reveals that ... is governed by ...
The asymmetry comes from ...

Intuitively, the statistic grows when the token is repeatedly observed in similar prefix environments,
since both the local and contextual factors reinforce one another.

The decomposition consists of three components: ...
We walk through the construction in three steps.
The term ... corresponds to ..., while ... corresponds to ...

We decompose the score into three parts:
\begin{align}
S &= S_{\text{local}} + S_{\text{context}} + S_{\text{position}},\\
S_{\text{local}} &= XBX^\top, \qquad
S_{\text{context}} = X\Psi^\top B^\top.
\end{align}
The first term captures immediate transitions, whereas the second term aggregates smoothed contextual evidence.

This observation leads to the following theorem.
Based on this characterization, we obtain ...
We validate this prediction empirically in Section ...
This decomposition suggests ..., which we formalize below.

This decomposition suggests a signed accumulation effect.
We formalize this below and validate it with targeted ablations in Section 4.

1. 动机句：为什么需要这条公式
2. 公式：用 equation 或 align 给出对象
3. 符号解释：每个符号在做什么
4. 直觉说明：为什么这个量合理
5. 过渡句：引到 theorem / claim / experiment

\paragraph{Motivation.}
To formalize the contextual dependence discussed above, we introduce a corpus statistic that aggregates prefix evidence.

\begin{equation}
\Psi_{ij} = \cdots
\end{equation}

Here, $\Psi_{ij}$ denotes ..., while $\mu_j$ acts as a centering term.
Intuitively, the statistic is large when token $e_j$ appears repeatedly in the contexts of $e_i$.

This characterization allows us to express the leading term of the value matrix as $\Psi^\top B^\top$,
which leads to the following theorem.

\documentclass[11pt,a4paper]{ctexart}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb,amsthm,mathtools,bm}
\usepackage{graphicx,booktabs,tabularx}
\usepackage{hyperref}
\usepackage{caption,subcaption}
\usepackage{listings}
\usepackage[most]{tcolorbox}
\usepackage[numbers]{natbib}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{definition}[theorem]{Definition}
\newtheorem{lemma}[theorem]{Lemma}

\paragraph{Motivation.}
To formalize the intuition above, we define ...

\begin{equation}
...
\end{equation}

Here, ... denotes ..., while ... represents ...

\begin{theorem}[Informal]
...
\end{theorem}

\paragraph{Interpretation.}
The theorem shows that ...

\paragraph{Localizing the failure.}
We first identify the unstable intermediate quantity ...

\begin{tcolorbox}[title=Claim]
...
\end{tcolorbox}

\begin{align}
...
\end{align}

This derivation shows that the update error is governed by ...

We validate this mechanism by ...