I believe this will be the last part of my MGF series, but we'll see. This is the MGF of a joint probability function (multiple variables), or a vector valued MGF. Originally I was going to use a proof that I put together myself, but it was long and cumbersome compared to a much shorter, comprehensible one I recently found. Firstly, I'll have to show a very useful result. Which is, if we have an expectation of $E[XY]$ where $X$ and $Y$ are random variables that are independent, then this is equivalent to $E[X]E[Y]$. To show this I'll have to explain a bit about independent events, and then move on to that result.
Now from an earlier post we know about the conditional probability, which is written as:
$$P(A|B)=\frac{P(A\cap B)}{P(B)}$$
Rearranged it becomes:
$$P(A|B)P(B)=P(A\cap B)$$
In other words, the probability that $A$ and $B$ happen is equal to the probability that $A$ given that $B$ has happened, multiplied by the probability of $B$. Now here's where we introduce the idea of independence. Let's say that we know $B$ has happened. Now what if $B$ happening doesn't affect $A$ at all? In other words, if we know $B$ happens, it doesn't change what $A$ could be. Let's say we flip a coin and get heads. Now let's say we flip a coin again. Does the fact we got heads on the first flip affect what we get on the second? Absolutely not. So, in mathematical notation, that's saying that $P(A|B)=P(A)$. Plugging that into our earlier function gives:
$$P(A)P(B)=P(A\cap B)$$
So the probability of $A$ and $B$ is the multiplication of both. This is known as the multiplication rule, and should make a lot of intuitive sense. What are the odds of getting two heads in a row? $(\frac{1}{2})(\frac{1}{2})=\frac{1}{4}$.
We can further the discussion by talking about PDFs of independent variables. Since a PDF is the probability of an event, independence should carry over naturally. So if we have two variables $X_{1}$ and $X_{2}$ that are independent, we know how conditional probability is defined:
$$f_{X_{2}|X_{1}}(X_{2}|X_{1})=\frac{f_{X_{2},X_{1}}(X_{2},X_{1})}{f_{X_{1}}(X_{1})}$$
Which, again, becomes:
$$f_{X_{2}|X_{1}}(X_{2}|X_{1})f_{X_{1}}(X_{1})=f_{X_{2},X_{1}}(X_{2},X_{1})$$
Now if like last time we can say that $X_{2}$ does not depend on $X_{1}$, then:
$$f_{X_{2}|X_{1}}(X_{2}|X_{1})=f_{X_2}(X_{2})$$
Which makes our previous equation:
$$f_{X_{2}}(X_{2})f_{X_{1}}(X_{1})=f_{X_{2},X_{1}}(X_{2},X_{1})$$
Which essentially says that the joint PDF of two independent variables is equal to the PDFs of both variables multiplied together. Now, we have enough information to find the result we were originally looking for. Let's start with $E[XY]$. Using the expectation operator this gives:
$$\int\int XYf_{X,Y}(X,Y)dxdy$$
But because the distributions are independent:
$$\int\int XYf_{X}(X)f_{Y}(Y)dxdy$$
Now we can separate the variables with the integrals. As we can see, we can pull out the PDF of $Y$ as well as $Y$ itself and integrate over $X$ and its distribution. Written out mathematically:
$$\int\int XYf_{X}(X)f_{Y}(Y)dxdy=\int Yf_{Y}(Y)\int Xf_{X}(X)dxdy$$
The inside integral becomes the average of $X$. Continuing:
$$ \int Y\mu_{X}f_{Y}(Y)dy=\mu_{X}\int Yf_{Y}(Y)dy=\mu_{X}\mu_{Y}=E[X]E[Y]$$
Which was the desired result. Now that we have this, we can talk about the MGF. The multiple variable MGF, where we take $n$ random variables, $X_{1},X_{2},...,X_{n}$, is defined as:
$$E[e^{t_{1}X_{1}+t_{2}X_{2}+...+t_{n}X_{n}}]=E[e^{\sum_{i=1}^{n}t_{i}X_{i}}]$$
Now we know that $e^{x+y}=e^{x}e^{y}$. Carrying this result gives us:
$$E[\prod_{i=1}^{n}e^{t_{i}X_{i}}]$$
Now we can finally use the result we proved earlier. Since this is the product of all the individual random variables, we can use independence. This gives us:
$$\prod_{i=1}^{n}E[e^{t_{i}X_{i}}]=\prod_{i=1}^{n}M_{X_{i}}$$
Where $M_{X_{i}}$ is the MGF of the $i$th variable. Hence, the vector valued MGF is simply the product of all the individual MGFs. From here we can use what we know about their individual MGFs to find their respective moments.
Friday, November 29, 2013
Monday, November 4, 2013
Negative Binomial and Geometric Distributions Part 1.
Hey everyone, I know it's been a while since I posted but I hope to
make it a bit more frequent in the future. I'll be making some changes
on the whole layout, but as for now I'll just continue with my series on
discrete distributions. This time I'll cover two distributions, the
negative binomial and the geometric, that are very related to each
other, as well as the two we've covered so far. Essentially every
distribution thus far is really a slight tweak of the binomial
distribution. For instance, here is the original binomial distribution
equation:
$$
B(n,p)=\left\{
\begin{array}{ll}
\left( {\begin{array}{*{20}c}
n \\ x \\
\end{array}} \right) P^{x}(1-P)^{n-x} \quad x=0,1,2,...,n\\
0,\quad \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
Where $n$ is the total number of trials, $P$ is the probability of success, $(1-P)$ the probability of failure, and $x$ the total number of successes. In an earlier post I explained the reasoning behind the term in front, but I could go over it again here. Let's say that you want to know the probability of getting two heads in a toss of four coins. Well, one way of doing this would be $P(1-P)P(1-P)$. A head on the first toss, a tail, another head, and then another tail. Is this the only way? Absolutely not. In fact, we could have had $PP(1-P)(1-P)$, or $P(1-P)(1-P)P$, and so on. The "Binomial Coefficient" in front counts all these different ways up, so you can properly measure the probability of getting two heads. Now, let's pose this question: What is the probability of getting $r$ successes on the $n+r$ trial? In other words, in $n+r-1$ trials we had exactly $r-1$ successes, and on the very next trial, $n+r$, we get a success. Let's have a variable, $x$, which denotes this total amount $n+r$. Therefore, total number of trials would be $x-1$ and total number of successes $r-1$ right before the final success. Before the final success, it would look like:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r-1}(1-P)^{x-r}$$
Now from our discussion before, we know that the probabilities can come in a different order when multiplying. However, in this scenario we know that the very last trial must be a success. Therefore, we must count up all the possible ways of getting $r-1$ successes, and then multiply that by the probability of getting the final success, $P$. This is essentially like taking our coin tossing experiment and only counting the terms with $P$ as the last term. This makes sense, since probabilities are multiplicative. If you want to know the probability of getting the $r$th success in $n+r$ trials, you multiply the probability of getting $r-1$ successes with the probability of getting another success, so $P$. That would look something like this:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r-1}(1-P)^{n}P$$
Where as you can see it's the probability of getting $r-1$ successes in $x-1$ trials, and the term on the end is the probability of getting the final success. This can be rearranged to get:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r}(1-P)^{n}$$
This is the negative binomial distribution. Unfortunately, there are different forms of the negative binomial distribution that are all essentially equivalent. This is the one I'll be using. Now what is the geometric distribution? The geometric distribution is the special case when $r=1$. This would become:
$$(1-P)^{n}P$$
I'll leave it to you to show that the binomial coefficient reduces to $1$. Now this makes sense. In the case that $r=1$ we have $n+r-1=n+1-1=n$. So we would have $n$ failures until the $n+1$ success. So in the sense that the Bernoulli distribution is a special case of the binomial, the geometric is a special case of the Negative binomial. Well, likewise, it would make sense that if we found the mean and variance of the negative binomial, it would include the mean and variance of the geometric distribution. The easiest way to find it would be using the MGF. To explain it in more depth I'll make
$$
B(n,p)=\left\{
\begin{array}{ll}
\left( {\begin{array}{*{20}c}
n \\ x \\
\end{array}} \right) P^{x}(1-P)^{n-x} \quad x=0,1,2,...,n\\
0,\quad \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
Where $n$ is the total number of trials, $P$ is the probability of success, $(1-P)$ the probability of failure, and $x$ the total number of successes. In an earlier post I explained the reasoning behind the term in front, but I could go over it again here. Let's say that you want to know the probability of getting two heads in a toss of four coins. Well, one way of doing this would be $P(1-P)P(1-P)$. A head on the first toss, a tail, another head, and then another tail. Is this the only way? Absolutely not. In fact, we could have had $PP(1-P)(1-P)$, or $P(1-P)(1-P)P$, and so on. The "Binomial Coefficient" in front counts all these different ways up, so you can properly measure the probability of getting two heads. Now, let's pose this question: What is the probability of getting $r$ successes on the $n+r$ trial? In other words, in $n+r-1$ trials we had exactly $r-1$ successes, and on the very next trial, $n+r$, we get a success. Let's have a variable, $x$, which denotes this total amount $n+r$. Therefore, total number of trials would be $x-1$ and total number of successes $r-1$ right before the final success. Before the final success, it would look like:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r-1}(1-P)^{x-r}$$
Now from our discussion before, we know that the probabilities can come in a different order when multiplying. However, in this scenario we know that the very last trial must be a success. Therefore, we must count up all the possible ways of getting $r-1$ successes, and then multiply that by the probability of getting the final success, $P$. This is essentially like taking our coin tossing experiment and only counting the terms with $P$ as the last term. This makes sense, since probabilities are multiplicative. If you want to know the probability of getting the $r$th success in $n+r$ trials, you multiply the probability of getting $r-1$ successes with the probability of getting another success, so $P$. That would look something like this:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r-1}(1-P)^{n}P$$
Where as you can see it's the probability of getting $r-1$ successes in $x-1$ trials, and the term on the end is the probability of getting the final success. This can be rearranged to get:
$$\left( {\begin{array}{*{20}c}x-1\\r-1\\ \end{array}}\right)P^{r}(1-P)^{n}$$
This is the negative binomial distribution. Unfortunately, there are different forms of the negative binomial distribution that are all essentially equivalent. This is the one I'll be using. Now what is the geometric distribution? The geometric distribution is the special case when $r=1$. This would become:
$$(1-P)^{n}P$$
I'll leave it to you to show that the binomial coefficient reduces to $1$. Now this makes sense. In the case that $r=1$ we have $n+r-1=n+1-1=n$. So we would have $n$ failures until the $n+1$ success. So in the sense that the Bernoulli distribution is a special case of the binomial, the geometric is a special case of the Negative binomial. Well, likewise, it would make sense that if we found the mean and variance of the negative binomial, it would include the mean and variance of the geometric distribution. The easiest way to find it would be using the MGF. To explain it in more depth I'll make
Friday, April 26, 2013
MGF of the Bernoulli and Binomial distributions.
Alright, a quick post about MGFs and the Bernoulli and Binomial distributions. Since the Bernoulli distribution is simply the Binomial distribution for the special case of $n=1$, we can see that the proof for the Binomial distribution will automatically include a proof for the Bernoulli distribution. I already showed it to find $E[x^2]$, but I'll restate it here. So, the MGF for discrete functions is:
$$\sum_{x=0}^{n}e^{tx}P_{x}(x)$$
Which for the Binomial distribution would be:
$$\sum_{x=0}^{n}e^{tx}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Where we do this:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (Pe^{t})^{x}(1-P)^{n-x}$$
Now, because of the Binomial theorem we know that:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (Pe^{t})^{x}(1-P)^{n-x}=(Pe^{t}+1-P)^{n}$$
This is a much more manageable form. Now we can take derivatives of this and get whatever moment we want. Let's use the general case of $n$, but then check it for $n=1$ (i.e. the Bernoulli distribution).
Well, (setting $(Pe^{t}+1-P)^{n}=M_{x}(t)$) the first derivative is:
$$\frac{d(M_{x})}{dt}=e^{t}Pn(e^{t}P+1-P)^{n-1}$$
By using the chain rule. So this is the first moment. Setting $t=0$ here will give us the mean. Let's check it:
$$e^{0}Pn(e^{0}P+1-P)^{n-1}=nP$$
Well, for the Binomial distribution we proved that the mean is $P$, and for the Binomial we showed it was $nP$, which is perfect. We have the Binomial mean, and if we set $n=1$ then we have the Bernoulli. Now for the second moment and the variance. To do this we can use the fact that $\sigma^{2}=E[x^2]-\mu^2$. Well, we know $\mu$, but we need to know $E[x^2]$, or the second moment. Taking the second derivative gives:
$$\frac{d^{2}(M_{x})}{dt^{2}}=e^{t}Pn(e^{t}P+1-P)^{n-1}+e^{t}Pn\left(e^{t}P(n-1)(e^{t}P+1-P)^{n-2}\right)$$
By use of the product rule. Setting $t=0$, taking note of the fact that $(P+1-P)=1$, and then simplifying gives us:
$$Pn+(P)^{2}n(n-1)=Pn+(Pn)^{2}-(P)^{2}n$$
Now, plugging this back into the variance formula, as well as the mean, gives us:
$$\sigma^{2}=Pn+(Pn)^{2}-(P)^{2}n-(Pn)^{2}=Pn-(P)^{2}n=Pn(1-P)$$
Which is the variance of the Binomial distribution. Setting $n=1$ gives us $P(1-P)$, which is exactly the variance for the Bernoulli distribution.
$$\sum_{x=0}^{n}e^{tx}P_{x}(x)$$
Which for the Binomial distribution would be:
$$\sum_{x=0}^{n}e^{tx}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Where we do this:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (Pe^{t})^{x}(1-P)^{n-x}$$
Now, because of the Binomial theorem we know that:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (Pe^{t})^{x}(1-P)^{n-x}=(Pe^{t}+1-P)^{n}$$
This is a much more manageable form. Now we can take derivatives of this and get whatever moment we want. Let's use the general case of $n$, but then check it for $n=1$ (i.e. the Bernoulli distribution).
Well, (setting $(Pe^{t}+1-P)^{n}=M_{x}(t)$) the first derivative is:
$$\frac{d(M_{x})}{dt}=e^{t}Pn(e^{t}P+1-P)^{n-1}$$
By using the chain rule. So this is the first moment. Setting $t=0$ here will give us the mean. Let's check it:
$$e^{0}Pn(e^{0}P+1-P)^{n-1}=nP$$
Well, for the Binomial distribution we proved that the mean is $P$, and for the Binomial we showed it was $nP$, which is perfect. We have the Binomial mean, and if we set $n=1$ then we have the Bernoulli. Now for the second moment and the variance. To do this we can use the fact that $\sigma^{2}=E[x^2]-\mu^2$. Well, we know $\mu$, but we need to know $E[x^2]$, or the second moment. Taking the second derivative gives:
$$\frac{d^{2}(M_{x})}{dt^{2}}=e^{t}Pn(e^{t}P+1-P)^{n-1}+e^{t}Pn\left(e^{t}P(n-1)(e^{t}P+1-P)^{n-2}\right)$$
By use of the product rule. Setting $t=0$, taking note of the fact that $(P+1-P)=1$, and then simplifying gives us:
$$Pn+(P)^{2}n(n-1)=Pn+(Pn)^{2}-(P)^{2}n$$
Now, plugging this back into the variance formula, as well as the mean, gives us:
$$\sigma^{2}=Pn+(Pn)^{2}-(P)^{2}n-(Pn)^{2}=Pn-(P)^{2}n=Pn(1-P)$$
Which is the variance of the Binomial distribution. Setting $n=1$ gives us $P(1-P)$, which is exactly the variance for the Bernoulli distribution.
Moment generating function Pt 2.
Alright, not really needed, but I'm going to throw in a few things that may come up. Essentially nifty little fun-facts. Let's start with the scenario
that you'd want a moment about the mean. To get that result we just start over with:
$$MGF\equiv\int_{-\infty}^{\infty}e^{tX}f(X)dx,~t\in\mathbb{R}$$
And replace $e^{tX}$ with $e^{t(X-\mu)}$. Now, the rest is pretty straight forward. We could sub in the expression for the expansion of $e$ and get:
$$e^{t(X-\mu)}=1+\frac{t(X-\mu)}{1!}+\frac{t^{2}(X-\mu)^{2}}{2!}+\frac{t^{3}(X-\mu)^{3}}{3!}+...$$
Following the steps of last time would give you essentially the same idea. However, there's another way we could go about it. Since we have $e^{t(X-\mu)}$, we can transform it into $e^{tX}e^{-t\mu}$. You'll notice the former is just what we had earlier, and the latter is a new term. But let's write it out:
$$\int_{-\infty}^{\infty}e^{-t\mu}e^{tX}f(X)dx$$
Well, the integral is just with respect to $X$, so we can treat $e^{-t\mu}$ as a constant and pull it out to get:
$$e^{-t\mu}\int_{-\infty}^{\infty}e^{tX}f(X)dx$$
Where the integral on the right is exactly the same as the MGF we had before. So this now becomes:
$$e^{-t\mu}M_{x}(t)$$
Differentiating and setting $t$ equal to zero like last time will give us the moments we want. To see this, let's try the first ones. We know the variance is defined as $E[(x-\mu)^2]$, and can alternatively be defined as $E[x^2]-\mu^2$. Well, this is the second moment about the mean, so let's try that in our alternate and simpler MGF. We'd have to take the second derivative and set it equal to zero. So let's start by differentiating it. Here:
$$ \frac{d(e^{-t\mu}M_{x}(t)}{dt}=M'_{x}(t)e^{-t\mu}+-\mu M_{x}(t)e^{-t\mu}$$
Now, this is the first moment about the mean. So it's essentially $E[x-\mu]=E[x]-E[\mu]=\mu-\mu=0$. Well, let's set $t=0$ to test that. We have:
$$M'_{x}(0)e^{0}+-\mu M_{0}(t)e^{0}=\mu(1)-\mu(1)(1)=\mu-\mu=0$$
Using the different moments of $x$ and the fact that $e^{0}=1$. The second derivative being:
$$\frac{d^{2}M_{x}(t)}{dt^2}=M''_{x}(t)e^{-t\mu}+-\mu M'_{x}(t)e^{-t\mu}+-\mu M'_{x}(t)e^{-t\mu}+\mu^{2} M_{x}(t)e^{-t\mu}$$
Setting $t=0$, and using the moments from the last post, we get:
$$M''_{x}(0)e^{0}+-2\mu M'_{x}(0)e^{0}+-\mu^{2} M'_{x}(0)e^{0}=E[x^2]+-2\mu^{2}+\mu^{2}=E[x^2]-\mu^{2}$$
Thus what we were trying to prove. Next post in this series will be about multiple variables. I may also mix the MGF series with the discrete distributions to get the MGFs of them.
$$MGF\equiv\int_{-\infty}^{\infty}e^{tX}f(X)dx,~t\in\mathbb{R}$$
And replace $e^{tX}$ with $e^{t(X-\mu)}$. Now, the rest is pretty straight forward. We could sub in the expression for the expansion of $e$ and get:
$$e^{t(X-\mu)}=1+\frac{t(X-\mu)}{1!}+\frac{t^{2}(X-\mu)^{2}}{2!}+\frac{t^{3}(X-\mu)^{3}}{3!}+...$$
Following the steps of last time would give you essentially the same idea. However, there's another way we could go about it. Since we have $e^{t(X-\mu)}$, we can transform it into $e^{tX}e^{-t\mu}$. You'll notice the former is just what we had earlier, and the latter is a new term. But let's write it out:
$$\int_{-\infty}^{\infty}e^{-t\mu}e^{tX}f(X)dx$$
Well, the integral is just with respect to $X$, so we can treat $e^{-t\mu}$ as a constant and pull it out to get:
$$e^{-t\mu}\int_{-\infty}^{\infty}e^{tX}f(X)dx$$
Where the integral on the right is exactly the same as the MGF we had before. So this now becomes:
$$e^{-t\mu}M_{x}(t)$$
Differentiating and setting $t$ equal to zero like last time will give us the moments we want. To see this, let's try the first ones. We know the variance is defined as $E[(x-\mu)^2]$, and can alternatively be defined as $E[x^2]-\mu^2$. Well, this is the second moment about the mean, so let's try that in our alternate and simpler MGF. We'd have to take the second derivative and set it equal to zero. So let's start by differentiating it. Here:
$$ \frac{d(e^{-t\mu}M_{x}(t)}{dt}=M'_{x}(t)e^{-t\mu}+-\mu M_{x}(t)e^{-t\mu}$$
Now, this is the first moment about the mean. So it's essentially $E[x-\mu]=E[x]-E[\mu]=\mu-\mu=0$. Well, let's set $t=0$ to test that. We have:
$$M'_{x}(0)e^{0}+-\mu M_{0}(t)e^{0}=\mu(1)-\mu(1)(1)=\mu-\mu=0$$
Using the different moments of $x$ and the fact that $e^{0}=1$. The second derivative being:
$$\frac{d^{2}M_{x}(t)}{dt^2}=M''_{x}(t)e^{-t\mu}+-\mu M'_{x}(t)e^{-t\mu}+-\mu M'_{x}(t)e^{-t\mu}+\mu^{2} M_{x}(t)e^{-t\mu}$$
Setting $t=0$, and using the moments from the last post, we get:
$$M''_{x}(0)e^{0}+-2\mu M'_{x}(0)e^{0}+-\mu^{2} M'_{x}(0)e^{0}=E[x^2]+-2\mu^{2}+\mu^{2}=E[x^2]-\mu^{2}$$
Thus what we were trying to prove. Next post in this series will be about multiple variables. I may also mix the MGF series with the discrete distributions to get the MGFs of them.
Monday, April 22, 2013
Discrete PMFs part 3. Variance of the Binomial distribution.
Alright, the variance of the binomial distribution. Now, we know that $\sigma^{2}=E[x^{2}]-\mu^{2}$, but we can check it with the binomial distribution if we'd like. So we have:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
And we also have that the mean, $\mu$, is $nP$. So, let's use our variance formula, which is $E[(x-\mu)^2]$. This becomes:
$$\sum_{x=0}^{n}(x-nP)^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Which simplifying the $(x-nP)^{2}$ term gives $(x^{2}-2xnP+[nP]^{2})$. Distributing out the probability function, and using the $\sum$ term as a linear operator gives:
$$\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}+2nP\sum_{0}^{n}x\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}+(nP)^{2}\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Where I was able to pull the $2nP$ out of the middle term since the summation doesn't affect it, as well as the $(nP)^2$ in the third term. This leaves it as $E[x]$, which we know is $nP$. Furthermore, the last term is simply the summation of the probability function, which my last post shows is $1$. So simplifying this down gives:
$$\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}-2(nP)^{2}+(nP)^{2}=\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}-(nP)^{2}$$
Which is easy to see that it's $E[x^{2}]-\mu^{2}$, thus verifying the variance formula for the binomial distribution. Now our job is to figure out what the $E[x^{2}]$ equals. Well, here's where we can use something we learned before. That's the MGF. This would look like:
$$\sum_{x=0}^{n}e^{tx}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
And we know that we'll need $M''(0)$ to get $E[x^2]$. But before we can worry about the second moment specifically, let's worry about getting the MGF in a form that's more manageable. So let's use a bit of footwork:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (e^{t}P)^{x}(1-P)^{n-x}=(e^{t}P+1-P)^{n}$$
By using the binomial theorem. Well, this is good, since now we can start taking derivatives. The first derivative is:
$$e^{t}Pn(e^{t}P+1-P)^{n-1}$$
By using the chain rule. The second we must use the product rule, which gives:
$$e^{t}Pn(e^{t}P+1-P)^{n-1}+e^{t}Pn\left(e^{t}P(n-1)(e^{t}P+1-P)^{n-2}\right)$$
Setting $t=0$ simplifies it to:
$$Pn+P^{2}n(n-1)$$
Plugging that back into our formula to find the variance gives:
$$\sigma^{2}=Pn+P^{2}n(n-1)-(nP)^{2}=Pn+P^{2}n^{2}-P^{2}n-P^{2}n^{2}=Pn-P^{2}n=Pn(1-P)$$
And now we have the variance in a manageable form. Next will be the Hypergeometric distribution.
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
And we also have that the mean, $\mu$, is $nP$. So, let's use our variance formula, which is $E[(x-\mu)^2]$. This becomes:
$$\sum_{x=0}^{n}(x-nP)^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Which simplifying the $(x-nP)^{2}$ term gives $(x^{2}-2xnP+[nP]^{2})$. Distributing out the probability function, and using the $\sum$ term as a linear operator gives:
$$\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}+2nP\sum_{0}^{n}x\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}+(nP)^{2}\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Where I was able to pull the $2nP$ out of the middle term since the summation doesn't affect it, as well as the $(nP)^2$ in the third term. This leaves it as $E[x]$, which we know is $nP$. Furthermore, the last term is simply the summation of the probability function, which my last post shows is $1$. So simplifying this down gives:
$$\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}-2(nP)^{2}+(nP)^{2}=\sum_{0}^{n}x^{2}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}-(nP)^{2}$$
Which is easy to see that it's $E[x^{2}]-\mu^{2}$, thus verifying the variance formula for the binomial distribution. Now our job is to figure out what the $E[x^{2}]$ equals. Well, here's where we can use something we learned before. That's the MGF. This would look like:
$$\sum_{x=0}^{n}e^{tx}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
And we know that we'll need $M''(0)$ to get $E[x^2]$. But before we can worry about the second moment specifically, let's worry about getting the MGF in a form that's more manageable. So let's use a bit of footwork:
$$\sum_{x=0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) (e^{t}P)^{x}(1-P)^{n-x}=(e^{t}P+1-P)^{n}$$
By using the binomial theorem. Well, this is good, since now we can start taking derivatives. The first derivative is:
$$e^{t}Pn(e^{t}P+1-P)^{n-1}$$
By using the chain rule. The second we must use the product rule, which gives:
$$e^{t}Pn(e^{t}P+1-P)^{n-1}+e^{t}Pn\left(e^{t}P(n-1)(e^{t}P+1-P)^{n-2}\right)$$
Setting $t=0$ simplifies it to:
$$Pn+P^{2}n(n-1)$$
Plugging that back into our formula to find the variance gives:
$$\sigma^{2}=Pn+P^{2}n(n-1)-(nP)^{2}=Pn+P^{2}n^{2}-P^{2}n-P^{2}n^{2}=Pn-P^{2}n=Pn(1-P)$$
And now we have the variance in a manageable form. Next will be the Hypergeometric distribution.
Wednesday, April 17, 2013
Discrete PMFs part 2. Binomial distribution. PMF proof and mean.
Now the binomial distribution. You'll see it's related to the Bernoulli for obvious reasons. It's defined as:
$$
B(n,p)=\left\{
\begin{array}{ll}
\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x} \quad x=0,1,2,...,n\\
0,\quad \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
Where:
$$\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)=\frac{{n!}}{{x!\left( {n - x} \right)!}}$$
Is the binomial coefficient. Before I explain the importance of it, let's worry about the other part. Now, the Bernoulli distribution was really just concerned with one success and one failure. However, what if we're worried about many successes and failures? Let's say we want to run four different trials? Well, that looks like:
$$P^{x}(1-P)^{4-x}$$
Where $x$ is the number of success, and hence $(4-x)$ is the number of failures. But as you'll notice, this isn't the actual probability. Let's say that there were two successes. This would look like:
$$(P)^{2}(1-P)^2=(P)(P)(1-P)(1-P)$$
But, notice that this is the probability of getting two successes in a row, and then two failures. Another option would be:
$$(P)(1-P)(P)(1-P)$$
This is the the same amount of successes and failures, but a completely different order. Since order does matter for us, this is where the binomial coefficient comes in. It counts up all the different orders in which we can get a certain number of successes and failures. So the full equation becomes:
$$\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Which is the probability of getting a certain number of successes in any order, multiplied the number of orders you can possibly get. For instance, flipping a coin twice and getting one head and one tail would be:
$$\left( {\begin{array}{*{20}c} 2 \\ 1 \\ \end{array}} \right) (\frac{1}{2})^{1}(1-\frac{1}{2})^{1}$$
Now, the binomial coefficient for these values equals:
$$\frac{{2!}}{{1!\left( {1} \right)!}}=\frac{2}{1}=2$$
And, multiplying this by the probability side gives:
$$(2)(\frac{1}{2})(\frac{1}{2})=(\frac{1}{2})$$
Which makes sense. The probability of getting a head then a tail in one order is $\frac{1}{4}$, yet there are two different ways to get it. Heads first, and heads second. So it becomes one half.
Alright, good stuff. But what about showing it's a probability function? It's obvious that it has $0\leq P$, since it's the same as the Bernoulli distribution (but this time with more successes and failures). But how do we know it adds up to one? Well, let's sum up all of the terms. That gives:
$$\sum_{0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Remember we're summing up the $x$ terms, since the number of trials is set. What varies is the number or successes and failures. So, we have:
$$(1-P)^{n}+nP(1-P)^{n-1}+...+nP^{n-1}(1-p)+P^{n}$$
The first term is the event of failing every trial, the next one is all the events of succeeding once and failing $n-1$ times, all the way up to succeeding $n$ times. Well, now we can use the binomial theorem, which states that:
$$(x+y)^{n}=\left( {\begin{array}{*{20}c} n \\ 0 \\ \end{array}} \right)x^{n}y^{0}+\left( {\begin{array}{*{20}c} n \\ 1 \\ \end{array}} \right)x^{n-1}y^{1}+...+\left( {\begin{array}{*{20}c} n \\ n-1 \\ \end{array}} \right)x^{1}y^{n-1}+\left( {\begin{array}{*{20}c} n \\ n \\ \end{array}} \right)x^{0}y^{n}$$
As you'll notice, if we use $y=1-P$ and $x=P$, this is exactly the same as our old equation. Well, reducing it to $(x+y)^n$ gives:
$$(x+y)^n=(P+1-P)^{n}=1^n=1$$
So this shows it's total probability is $1$, which is what we were trying to prove. Now what about the expectation and variance? Let's start with the expectation. That is:
$$\sum_{0}^{n}x\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}$$
Writing it out gives:
$$(0)\left( {\begin{array}{*{20}c} n \\ 0 \\ \end{array}} \right)P^{0}(1-P)^{n}+(1)\left( {\begin{array}{*{20}c} n \\ 1 \\ \end{array}} \right)P^{1}(1-P)^{n-1}+...+\left( {\begin{array}{*{20}c} n \\ n-1 \\ \end{array}} \right)P^{n-1}(1-P)^{1}+\left( {\begin{array}{*{20}c} n \\ n \\ \end{array}} \right)P^{n}(1-P)^{0}$$
Now, we can use this fact:
$$\frac{x(n)!}{(x)!(n-x)!}=\frac{(n)!}{(x-1)!(n-x)!}=n\left[\frac{(n-1)!}{(x-1)!(n-x)!}\right]$$
Now, the denominator of the fraction can be changed to $(x-1)!([n-1]-[x-1])!$, since simplifying the latter factorial still gives $(n-x)!$. Alright, well using this with our previous equation (since this is the coefficient of all of these), and simplifying a bit, we get:
$$nP\left[(1-P)^{n-1}+(n-1)P(1-P)^{n-2}+...+(n-1)P^{n-2}(1-P)+P^{n-1}\right]$$
Now, we can see the inside is the binomial coefficent:
$$\sum_{y=0}^{n-1}\left( {\begin{array}{*{20}c} n-1 \\ y \\ \end{array}} \right) P^{y}(1-P)^{n-1-y}=(P+(1-P))^{n-1}=(1)^{n-1}=1$$
Where $y=x-1$. So the equation reduces to $nP$, which is the mean.
For variance we have I'll save it for another blog post.
$$
B(n,p)=\left\{
\begin{array}{ll}
\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x} \quad x=0,1,2,...,n\\
0,\quad \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
Where:
$$\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)=\frac{{n!}}{{x!\left( {n - x} \right)!}}$$
Is the binomial coefficient. Before I explain the importance of it, let's worry about the other part. Now, the Bernoulli distribution was really just concerned with one success and one failure. However, what if we're worried about many successes and failures? Let's say we want to run four different trials? Well, that looks like:
$$P^{x}(1-P)^{4-x}$$
Where $x$ is the number of success, and hence $(4-x)$ is the number of failures. But as you'll notice, this isn't the actual probability. Let's say that there were two successes. This would look like:
$$(P)^{2}(1-P)^2=(P)(P)(1-P)(1-P)$$
But, notice that this is the probability of getting two successes in a row, and then two failures. Another option would be:
$$(P)(1-P)(P)(1-P)$$
This is the the same amount of successes and failures, but a completely different order. Since order does matter for us, this is where the binomial coefficient comes in. It counts up all the different orders in which we can get a certain number of successes and failures. So the full equation becomes:
$$\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Which is the probability of getting a certain number of successes in any order, multiplied the number of orders you can possibly get. For instance, flipping a coin twice and getting one head and one tail would be:
$$\left( {\begin{array}{*{20}c} 2 \\ 1 \\ \end{array}} \right) (\frac{1}{2})^{1}(1-\frac{1}{2})^{1}$$
Now, the binomial coefficient for these values equals:
$$\frac{{2!}}{{1!\left( {1} \right)!}}=\frac{2}{1}=2$$
And, multiplying this by the probability side gives:
$$(2)(\frac{1}{2})(\frac{1}{2})=(\frac{1}{2})$$
Which makes sense. The probability of getting a head then a tail in one order is $\frac{1}{4}$, yet there are two different ways to get it. Heads first, and heads second. So it becomes one half.
Alright, good stuff. But what about showing it's a probability function? It's obvious that it has $0\leq P$, since it's the same as the Bernoulli distribution (but this time with more successes and failures). But how do we know it adds up to one? Well, let's sum up all of the terms. That gives:
$$\sum_{0}^{n}\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right) P^{x}(1-P)^{n-x}$$
Remember we're summing up the $x$ terms, since the number of trials is set. What varies is the number or successes and failures. So, we have:
$$(1-P)^{n}+nP(1-P)^{n-1}+...+nP^{n-1}(1-p)+P^{n}$$
The first term is the event of failing every trial, the next one is all the events of succeeding once and failing $n-1$ times, all the way up to succeeding $n$ times. Well, now we can use the binomial theorem, which states that:
$$(x+y)^{n}=\left( {\begin{array}{*{20}c} n \\ 0 \\ \end{array}} \right)x^{n}y^{0}+\left( {\begin{array}{*{20}c} n \\ 1 \\ \end{array}} \right)x^{n-1}y^{1}+...+\left( {\begin{array}{*{20}c} n \\ n-1 \\ \end{array}} \right)x^{1}y^{n-1}+\left( {\begin{array}{*{20}c} n \\ n \\ \end{array}} \right)x^{0}y^{n}$$
As you'll notice, if we use $y=1-P$ and $x=P$, this is exactly the same as our old equation. Well, reducing it to $(x+y)^n$ gives:
$$(x+y)^n=(P+1-P)^{n}=1^n=1$$
So this shows it's total probability is $1$, which is what we were trying to prove. Now what about the expectation and variance? Let's start with the expectation. That is:
$$\sum_{0}^{n}x\left( {\begin{array}{*{20}c} n \\ x \\ \end{array}} \right)P^{x}(1-P)^{n-x}$$
Writing it out gives:
$$(0)\left( {\begin{array}{*{20}c} n \\ 0 \\ \end{array}} \right)P^{0}(1-P)^{n}+(1)\left( {\begin{array}{*{20}c} n \\ 1 \\ \end{array}} \right)P^{1}(1-P)^{n-1}+...+\left( {\begin{array}{*{20}c} n \\ n-1 \\ \end{array}} \right)P^{n-1}(1-P)^{1}+\left( {\begin{array}{*{20}c} n \\ n \\ \end{array}} \right)P^{n}(1-P)^{0}$$
Now, we can use this fact:
$$\frac{x(n)!}{(x)!(n-x)!}=\frac{(n)!}{(x-1)!(n-x)!}=n\left[\frac{(n-1)!}{(x-1)!(n-x)!}\right]$$
Now, the denominator of the fraction can be changed to $(x-1)!([n-1]-[x-1])!$, since simplifying the latter factorial still gives $(n-x)!$. Alright, well using this with our previous equation (since this is the coefficient of all of these), and simplifying a bit, we get:
$$nP\left[(1-P)^{n-1}+(n-1)P(1-P)^{n-2}+...+(n-1)P^{n-2}(1-P)+P^{n-1}\right]$$
Now, we can see the inside is the binomial coefficent:
$$\sum_{y=0}^{n-1}\left( {\begin{array}{*{20}c} n-1 \\ y \\ \end{array}} \right) P^{y}(1-P)^{n-1-y}=(P+(1-P))^{n-1}=(1)^{n-1}=1$$
Where $y=x-1$. So the equation reduces to $nP$, which is the mean.
For variance we have I'll save it for another blog post.
Tuesday, April 16, 2013
Discrete PMFs part 1. Bernoulli distribution.
Alright, I realize I still have to do that MGF post, but I also realized most of my stuff has dealt with continuous distributions. A lot of the proofs are pretty easy (just switch to summation notation), but I figured I'd do a post or two covering the main discrete distributions.
$$
P_{x}(x)=\left\{
\begin{array}{ll}
P^{x}(1-P)^{1-x} \quad x=0,1\\
0 \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
This can be considered a success and failure distribution. For instance, if there are two mutually exclusive events, success=$1$ and failure=$0$, with a probability $P$ such that $0\leq P\leq 1$, then the probability of success is $P$, and failure is $(1-P)$. To see that, let's say it succeeds:
$$P^{1}(1-P)^{1-1}=P^{1}(1-P)^{0}=P(1)=P$$
Since we use the fact that $(1-P)$ raised to $0$ is $1$. For failure:
$$P^{0}(1-P)^{1-0}=(1)(1-P)=(1-P)$$
Using the fact that $P^0$ equals $1$. So, we've verified that the probability of success is $P$, and failure $(1-P)$. But, this is really just the beginning. How do we know that it's a probability function? Well, there are two conditions it must satisfy. Since $0\leq P$, we know that the probability of success is greater than or equal to zero. Furthermore, since $P\leq1$ we know that $(1-P)$ must be greater than or equal to $0$, so we know that both events have a probability greater than or equal to $0$. Now, since they're independent and discrete their total probability is equal to:
$$\sum_{x}P_{x}(x)$$
Which, in this case is:
$$\sum_{0}^{1}P_{x}(x)=P_{x}(0)+P_{x}(1)=P+(1-P)=1$$
So this shows it's a PMF. What about it's CMF? That's set as:
$$F(x)=P[X\leq x]=\sum_{x_{i}\leq x}P_{x}(x_{i})$$
Where:
$$\lim_{x\rightarrow-\infty}F(x)=0$$
And,
$$\lim_{x\rightarrow\infty}F(x)=1$$
Something worth covering is the expectation and the variance. For discrete functions the expected value is defined as:
$$\sum_{i=1}^{\infty}x_{i}P_{i}$$
For this case, it's:
$$\sum_{i=0}^{1}x P^{x}(1-P)^{1-x}=(0) P^{0}(1-P)^{1}+(1)P^{1}(1-P)^{0}=P$$
And the variance, defined as:
$$\sum_{i=1}^{\infty}(x_{i}-\mu)^{2}P_{i}$$
Which for the Bernoulli distribution is:
$$\sum_{0}^{1}(x-P)^{2}P^{x}(1-P)^{1-x}=(P)^{2}(1-P)+(1-P)^{2}P$$
Where we can pull out a $P$ and $(1-P)$ to get:
$$P[P+(1-P)](1-P)=P(1)(1-P)=P(1-P)$$
That was a lot of the basic stuff for the Bernoulli distribution. I'll cover the binomial one next since it's so closely related.
$$
P_{x}(x)=\left\{
\begin{array}{ll}
P^{x}(1-P)^{1-x} \quad x=0,1\\
0 \qquad \qquad \qquad otherwise
\end{array}
\right.
$$
This can be considered a success and failure distribution. For instance, if there are two mutually exclusive events, success=$1$ and failure=$0$, with a probability $P$ such that $0\leq P\leq 1$, then the probability of success is $P$, and failure is $(1-P)$. To see that, let's say it succeeds:
$$P^{1}(1-P)^{1-1}=P^{1}(1-P)^{0}=P(1)=P$$
Since we use the fact that $(1-P)$ raised to $0$ is $1$. For failure:
$$P^{0}(1-P)^{1-0}=(1)(1-P)=(1-P)$$
Using the fact that $P^0$ equals $1$. So, we've verified that the probability of success is $P$, and failure $(1-P)$. But, this is really just the beginning. How do we know that it's a probability function? Well, there are two conditions it must satisfy. Since $0\leq P$, we know that the probability of success is greater than or equal to zero. Furthermore, since $P\leq1$ we know that $(1-P)$ must be greater than or equal to $0$, so we know that both events have a probability greater than or equal to $0$. Now, since they're independent and discrete their total probability is equal to:
$$\sum_{x}P_{x}(x)$$
Which, in this case is:
$$\sum_{0}^{1}P_{x}(x)=P_{x}(0)+P_{x}(1)=P+(1-P)=1$$
So this shows it's a PMF. What about it's CMF? That's set as:
$$F(x)=P[X\leq x]=\sum_{x_{i}\leq x}P_{x}(x_{i})$$
Where:
$$\lim_{x\rightarrow-\infty}F(x)=0$$
And,
$$\lim_{x\rightarrow\infty}F(x)=1$$
Something worth covering is the expectation and the variance. For discrete functions the expected value is defined as:
$$\sum_{i=1}^{\infty}x_{i}P_{i}$$
For this case, it's:
$$\sum_{i=0}^{1}x P^{x}(1-P)^{1-x}=(0) P^{0}(1-P)^{1}+(1)P^{1}(1-P)^{0}=P$$
And the variance, defined as:
$$\sum_{i=1}^{\infty}(x_{i}-\mu)^{2}P_{i}$$
Which for the Bernoulli distribution is:
$$\sum_{0}^{1}(x-P)^{2}P^{x}(1-P)^{1-x}=(P)^{2}(1-P)+(1-P)^{2}P$$
Where we can pull out a $P$ and $(1-P)$ to get:
$$P[P+(1-P)](1-P)=P(1)(1-P)=P(1-P)$$
That was a lot of the basic stuff for the Bernoulli distribution. I'll cover the binomial one next since it's so closely related.
Subscribe to:
Posts (Atom)