cointypes <- c(0.3, 0.4, 0.5, 0.6, 0.7) # Probability of headsBayesian Statistics
4. On Bayes’ Theorem
Probabilities? It’s probably ambiguous
- Despite its ubiquity, the concept of probability is actually quite ambiguous
- Classical meaning:
- \(P(event~A) = \frac{number\ of\ of\ events\ A}{number\ of\ possible\ events} = \frac{4}{8} = 0.5\)
- This formula only works when there is a finite number of equiprobable outcomes
- What is the probability of raining tomorrow?
- \(P(rain) = \frac{rain}{\{rain,~no~rain\}} = \frac{1}{2}\)
- That doesn’t seem right…
- Typical conceptualization of probabilities applies to series, not to single events
- But what about non-repeatable events? What is the probability of you learning something new during this course?
Thomas Bayes
- \(P(A|B) = \frac{P(A)~P(B|A)}{P(B)}\)
- Thomas Bayes (1701-1761) was an English statistician, philosopher and Presbyterian minister
- He is best known for his work in probability theory
- Bayes never actually published what would become his most famous accomplishment; his notes were edited and published posthumously
- Ironically, he is not the “biggest” name in the field of Bayesian statistics. It is later authors who applied his ideas to develop a new statistical framework
Only portrait Thomas Bayes. But probably not him.
The Cop, the Gentleman and the Sketchy Looking Guy
Likelihood Model
- A likelihood model is a model that links predictors with outcomes (a predictor variable value of X generates a dependent variable value of y)
- The likelihood of an event is the probability of that event according to various values of the predictor(s) following the likelihood model
- What is the likelihood of obtaining heads given various possibilities (types of coin)?
- Corresponds to probability of a “heads” outcome given each value of the parameter space (here, 5 discrete values of coin type)
likelihood_heads <- c(0.3, 0.4, 0.5, 0.6, 0.7) # Likelihood is the same
Priors
- What is our prior belief about the probability of each type of coin being used?
- The Gentleman is attractive and looks very honest
- We assign a high prior probability to the coin being fair (“3”)
- It is also fairly possible that the coin is slightly imperfect (“2”)
- Given how honest the man looks, we think it’s unlikely that the coin is rigged (“1”)
# cointypes: [0.3, 0.4, 0.5, 0.6, 0.7]
prior <- c(1, 2, 3, 2, 1) # We assign "weights" to each coin type
prior <- prior / sum(prior) # Normalize it so that the total = 1
prior[1] 0.1111111 0.2222222 0.3333333 0.2222222 0.1111111
Apply Bayes’s Theorem
- The posterior is proportional to the prior and the likelihood (note that we ignore the denominator, also referred to as “the constant of proportionality” by normalizing so that the probabilities sum to 1)
# Posterior probabilities given the head
posterior <- prior * likelihood_heads
posterior <- posterior / sum(posterior)
posterior[1] 0.06666667 0.17777778 0.33333333 0.26666667 0.15555556- The probabilities represent our updated beliefs after seeing the data (one coin toss)
Second Toss
- It’s a head again!
- We can update our posterior probabilities again but this time, by starting taking the previous posterior as new prior
prior <- posterior
# Update
posterior <- prior * likelihood_heads
posterior <- posterior / sum(posterior)
Third Toss
- Third toss, it’s heads again!
- Compute the new posterior probabilities
- What can we conclude?
prior <- posterior
# Update
posterior <- prior * likelihood_heads
posterior <- posterior / sum(posterior)
Sequential Evidence Accumulation
- How many consecutive tails should we observe to change our beliefs to favour the biased (0.3) coin?
likelihood_tails <- c(0.7, 0.6, 0.5, 0.4, 0.3)
for(n in 4:10) {
posterior <- posterior * likelihood_tails
posterior <- posterior / sum(posterior)
}
- We can see how the evidence accumulates over time depending on the data (and of our priors)
Back to the problem
- Let’s go back to our original problem. We want to know if the coin is fair or not, but we don’t have 20 tosses to spare.
- We have limited time and resources, so we need to make a decision as soon as possible
- Let’s say we have only 3 tosses, and all of them are “heads” (as before)
- After 3 tosses, we conclude that the coin is most likely the Imperfect (0.6), which is fine, we won’t put him to prison
Different Priors
- The likelihood is the same as before
- But we have different prior beliefs is different
prior <- c(3, 2, 1, 2, 3) # We assign "weights" to each coin type
prior <- prior / sum(prior) # Normalize it so that the total = 1
prior[1] 0.27272727 0.18181818 0.09090909 0.18181818 0.27272727
Update posterior
- We can update the posterior after seeing 3 heads in a row
for(n in 1:3) {
posterior <- prior * likelihood_heads
posterior <- posterior / sum(posterior)
prior <- posterior
}
- We can see that the posterior is different from the previous case
- We conclude that the coin is most likely the Biased (0.)
- In any case, we put the sketchy guy in jail. Thanks, maths!
What? I can tweak my results by tweaking the prior? It feels illegal
- Yes, the results depend on the priors. And it’s a good thing (we’ll come back to that)
- Priors are not necessarily subjective
- Results converge to the same posterior with enough data (if the data is clear, the results are clear regardless of the prior)
The End (for now)
Thank you!
