Recall that the Law of Large Numbers implies that sample mean converges to the popultion mean. This holds true regardless of the underlying distribution of the population. Figure 1 illustrates the conconvergence for normal, Bernoulli, and exponential random variables.
Figure 1 shows that the sampling distributions look similar also in shape between the three underlying distributions when the sample size is large. The sampling distributions are converging to a common shape.
Let \(Y_1\)
, \(Y_2\)
, … be random variables. Then we say that the sequence ${Y_n} converges in distribution to \(Y\)
, if for all \(y\in\mathbb{R}\)
such that \(P(Y=y)=0\)
, we have
$$\lim_{n\to\infty}P\left(Y_n\le y\right) = P\left(Y\le y\right)$$
and we write
$$Y_n\overset{d}{\longrightarrow}Y.$$
In other words, the distribution of \(Y_n\)
becomes closer and closer to that of \(Y\)
.
Recall a Poisson random variable is the count of events among an infinite number of independent and identical Bernoulli trials in a fixed interval.
For a finite \(n\)
number of intervals, we can use \(X_n \sim \text{Binom}\left(n, p\right)\)
to describe the count.
Let \(\lambda = np\)
. Then, the probability mass function of \(X_n\)
is given by
$$p_{X_n}(x;\lambda)=\binom{n}{x}\left(\frac{\lambda}{n}\right)^x\left(1-\frac{\lambda}{n}\right)^{n-x}=\frac{n!}{x!\left(n-x\right)!}\left(\frac{\lambda}{n}\right)^x\left(1-\frac{\lambda}{n}\right)^{n-x}.$$
Taking \(n\to\infty\)
, we can show that
$$\lim_{n\to\infty}p_{X_n}\left(x;\lambda\right)=\frac{\lambda^x e^{-\lambda}}{x!},$$
or \(X_n \overset{d}{\longrightarrow}Y\)
where \(Y\sim\text{Pois}(\lambda)\)
.
Figure 1 suggests that the distribution of \(\overline{X}_n\)
for the three underlying distributions investigated. The Central Limit Theorem explains which distribution they are converging to.
Let \(X_1\)
, \(X_2\)
, … be independent and identically distributed random variables with finite expectation \(\mu\)
and finite positive variance \(\sigma^2\)
. For \(n\ge 1\)
, let
$$Z_n=\frac{\sqrt{n}\left(\overline{X}_n - \mu\right)}{\sigma}$$
then for any number \(a\)
,
$$\lim_{n\to\infty} P(Z_n\le a) = \Phi(a),$$
where \(\Phi\)
is the cumulative distribution function of the standard normal distribution. In other words,
$$\frac{\sqrt{n}\left(\overline{X}_n - \mu\right)}{\sigma} \overset{d}{\longrightarrow} Z,$$
where \(Z\sim N(0,1)\)
.
We can rewrite the limit in terms of \(\overline{X}_n\)
.
$$\lim_{n\to\infty} P\left(Z_n\le a\right) = \Phi(a)$$
$$\implies\lim_{n\to\infty} P\left(\frac{\sqrt{n}\left(\overline{X}_n - \mu\right)}{\sigma}\le a\right) = \Phi(a)$$
$$\implies\lim_{n\to\infty} P\left(\overline{X}_n\le a\frac{\sigma}{\sqrt{n}}+\mu\right) = \Phi(a)$$
We say \(\overline{X}_n\)
is asymptotically normal as it resembles a normal random variable in the limit. In practice, when \(n\)
is sufficiently large, \(\overline{X}_n\)
is approximately normal with mean \(\mu\)
and variance \(\left.\sigma^2\right/\sqrt{n}\)
.