I’ve come across somebody who has what appears to be a rather odd version of decision analysis that looks like it is based on a gross misunderstanding of Bayesian analysis.
He has claimed that, for any decision that has only two possible outcomes (essentially, “signal” or “no signal”), and all current data has not been "no signal), but only a very small portion of a population is surveyed, one is compelled and has no choice but to presume that the prior probability of “signal” must and can only be 50%.
His specific issue is looking for extraterrestrial life. His “reasoning” is that, since we have nothing to go on for any sort of life associated with other stars, we must presume that there is a 50% chance that extraterrestrial life exists until we learn otherwise. His reasoning is, since there are only two possible outcomes, without any sort of information, we are forced by “logic” to assign equal prior probability to them. He then starts to natter on about tossing coins, ignoring the possibility that there might be something to the intentionally symmetrical construction of coins that could influence probability behavior unique to coin-like objects.
Now, as I understand Bayesian analysis, it only requires that one have some sort of prior probability but the theory specifies no necessary value. I could set the prior probability at “zero” or even “quite small” without necessarily specifying a fixed numerical value.
So, do I understand Bayesian theory correctly, or does it demand that, given m possible outcomes and no information, prior probability must, under all circumstances, be 1/m?
If you have no information whatsoever, then it’s perfectly reasonable to assume any values for the priors. Your friend is referring to the principle of indifference, which is a rule of thumb stating that, if all possible priors are as good as any other, you should choose equally likely outcomes.
AFAIK, that’s generally well-regarded, but it leads to a rather famous paradox. Suppose I tell you that the ratio of water to wine in a jug is somewhere between 1/3 and 3, and ask you to compute the probability that it’s between 1/2 and 2. You assume that the ratio of water to wine is uniformly distributed, and calculate that the probability is 9/16. All’s well and good.
But wait a minute: what reason did you have to choose the ratio of water to wine as the uniformly distributed variable? Why not the ratio of wine to water? That leads to quite a different outcome (7/16, if I’m not mistaken).
The thing to remember about Bayesian analysis is that your output is only as good as your input. Start with good input, and you’ll get good output. Start with bad input…well, maybe you shouldn’t bet with the output.
At what point does one reject this principle. Suppose I have looked at 100 stars and found nothing. Does that then favor a lower prior probability than 1/2? He claims that, since I’ve yet found no evidence, I cannot adjust my probability. Until I find extraterrestrial life and then divide that one by the number of stars surveyed, the probability, according to him, must remain at 1/2.
That sounds just plain screwy to me. Wouldn’t Bayesian analysis say that repeated negative measures would require revising prior probability lower and lower before conducting a new experiment? Indeed, isn’t that supposed to be one of the strengths of Bayesian analysis, that it provides a rigorous way to approach “X is not the case” by means of developing successively lower probability?
He’s a screwball. Your expirement fits the binomial model, so if the probability is 1/2, the probability of getting 0 matches in 100 tries is 1/2[sup]100[/sup] (roughly 1/10[sup]30[/sup]). That’s an untenable claim.
In making your inference about the next star’s life-having status, you have secreted introduced a model and assigned it a high prior probablity. Your model says that all stars are, in some sense, the same, and that observations of one teach you things about those you haven’t seen. Taking a specific form of such a model:
“Each extra-Sol star in the universe has probability p of having life around it.”
(Of course, your gut-feeling model doesn’t speak in such terms, but anyway…) This model allows you to say some specific things about the priors you should apply to the 101st star. However, you must now apply prior probabilities to the complete set of models! In practice, you are clumping all models of the type “stars behave similarly” into one category and assigning that category a large probability of being The Way Thing Are. That is not invalid if such priors fit your beliefs, but (echoing ultrafilter’s comment), your answer will only be as good as those (input) beliefs.
The point is that you are trying to do Bayesian inference. You don’t know the true behavior – you are trying to infer it. Perhaps the model is binomial. Perhaps it is Polya. Perhaps it is such that “every 101st star that earth-based humans observe will have life on it.” (To be sure, I would personally assign a low prior probability to the last model there, but I am introducing an actual human bias by doing to, and I need to be aware of it.)
Are we talking about the probability of finding life in a specific place we look? My gut feeling tells me that’s a very small number. And the fact that we’ve looked in a couple hundred places and haven’t seen it yet certainly agrees with that gut feeling.
Or, are we talking about the probability that there is life somewhere? My gut feeling is that that is the very small number mentioned above, multiplied by a very large number of specific places we could look. And in the absence of more data, it would indeed be foolish to claim we must assign exactly 0.5 to this; all we can really say is that it’s somewhere between 0 and 1. The fact that we’ve looked in a few places, and haven’t found it yet, doesn’t really give much additional information on this score.