Central Limit Theorem

Preamble and Introduction

From an engineering perspective, sequences of random variables converging to a known distribution helps in desigining efficient systems.

Define $S_n = x_1 + x_2 + \cdots + x_n$ as the sequence sum of independent and identically distributed (i.i.d.) random variables with mean $\mathbb{E}[x]$ and variance $\sigma^2$.

Law of Large Numbers

For a sequence of i.i.d. random variables $x_1, x_2, \cdots$. For $n \leq 1$ we have, $$ S_n = \sum\limits_{i=1}^n x_n $$

Then the law of large numbers gives, $$ \lim\limits_{n \rightarrow \infty}\frac{S_n}{n} = \lim\limits_{n \rightarrow \infty}\frac{1}{n} \sum\limits_{i=1}^n x_n = \mathbb{E}[x] $$

Meaning the sequence of empirical averages $\frac{S_n}{n}$ converges asymptotically to the first moment $\mathbb{E}[x]$.

Central Limit Theorem

Theorem states that the distribution of a sequence of random variable $S_n$ converges asymptotically to the normal distribution. More formally, let $x_1, x_2, \cdots$ be the i.i.d. random variables each with mean $\mathbb{E}[x]$ and variance $\sigma^2$.

For any $x \in \mathbb{R}$, the distribution, $$ \frac{S_n - n\mathbb{E}[x]}{\sigma \sqrt{n}} $$

converges to the normal distribution. Asymptotically we have, $$ \lim\limits_{n \rightarrow \infty} Pr\bigg(\frac{S_n - n\mathbb{E}[x]}{\sigma \sqrt{n}} \leq x\bigg) = \int_{-\infty}^{x} \frac{1}{\sqrt{2\pi}} e^{\frac{-\tau^2}{2}} d\tau $$

For a sufficiently large $n$, the cumulitive density function (CDF) of $S_n$ as, $$ F_{S_n}(x) = Pr(S_n \leq x) = Pr\Bigg(\frac{S_n - n\mathbb{E}[x]}{\sigma\sqrt{n}} \leq \frac{x - n\mathbb{E}[x]}{\sigma\sqrt{n}}\Bigg) $$ $$ \approx \Phi\Bigg(\frac{x - n\mathbb{E}[x]}{\sigma\sqrt{n}}\Bigg) $$

Where $\Phi(\cdot)$ is the CDF of a normal random variable.