1.3. Checking the sum and product rules, and their consequences¶
Goal: Check using a very simple example that the Bayesian rules are consistent with standard probabilities based on frequencies. Also check notation and vocabulary.
Bayesian rules of probability as principles of logic¶
Notation: \(p(x \mid I)\) is the probability (or pdf) of \(x\) being true given information \(I\)
Sum rule: If set \(\{x_i\}\) is exhaustive and exclusive,
\[ \sum_i p(x_i \mid I) = 1 \quad \longrightarrow \quad \color{red}{\int\!dx\, p(x \mid I) = 1} \]cf. complete and orthonormal
implies marginalization (cf. inserting complete set of states or integrating out variables - but be careful!)
\[ p(x \mid I) = \sum_j p(x,y_j \mid I) \quad \longrightarrow \quad \color{red}{p(x \mid I) = \int\!dy\, p(x,y \mid I)} \]Product rule: expanding a joint probability of \(x\) and \(y\)
\[ \color{red}{ p(x,y \mid I) = p(x \mid y,I)\,p(y \mid I) = p(y \mid x,I)\,p(x \mid I)} \]If \(x\) and \(y\) are mutually independent: \(p(x \mid y,I) = p(x \mid I)\), then
\[ p(x,y \mid I) \longrightarrow p(x \mid I)\,p(y \mid I) \]Rearranging the second equality yields Bayes’ Rule (or Theorem)
\[ \color{blue}{p(x \mid y,I) = \frac{p(y \mid x,I)\, p(x \mid I)}{p(y \mid I)}} \]
See Cox for the proof.
Answer the questions in italics.¶
Check answers with your neighbors. Ask for help if you get stuck or are unsure.
%%html
<style>
table { width:80% !important; }
table td, th { border: 1px solid black !important;
text-align:center !important;
font-size: 20px }
</style>
TABLE 1 |
Blue |
Brown |
Total |
|---|---|---|---|
Tall |
1 |
17 |
18 |
Short |
37 |
20 |
57 |
Total |
38 |
37 |
75 |
TABLE 2 |
Blue |
Brown |
Total |
|---|---|---|---|
Tall |
|
|
|
Short |
|
|
|
Total |
|
|
|
Table 1 shows the number of blue- or brown-eyed and tall or short individuals in a population of 75.
Fill in the blanks in Table 2 with probabilities (in decimals with three places, not fractions) based on the usual “frequentist” interpretations of probability (which would say that the probability of randomly drawing an ace from a deck of cards is 4/52 = 1/13). Add x’s in the row and/or column that illustrates the sum rule.
What is \(p(short, blue)\)? Is this a joint or conditional probability? What is \(p(blue)\)?
From the product rule, what is \(p(short | blue)\)? Can you read this result directly from the table?
Apply Bayes’ theorem to find \(p(blue | short)\) from your answers to the last part.
What rule does the second row (the one starting with “Short”) illustrate? Write it out in \(p(\cdot)\) notation.
Are the probabilities of being tall and having brown eyes mutually independent? Why or why not?