It’s a psychological problem. Every specific sequence has the same probability, but in our mind we lump all sequences that show no recognizable pattern into one category: “random sequence”, which comprises the vast majority of sequences. So in our mind we compare the probability of hitting one specific sequence to the probability of hitting any one of a very large group of sequences.
…
Thanks guys, you’ve jogged my mind on this.
As many of you said, it’s not the low probability of an event that makes it rightly surprising, it’s how low the probability of it is as compared to the probability of other events, where “events” here aren’t single possibilities but entire partitions of the class of all possible events.
And of course “rightly surprising” assumes there’s some “right” way to partition that class, which is where you can get into trouble. In the heads/tails case though it’s natural to consider the (colloquially speaking) “random looking” sequences as a sensible partition, since they are the ones naturally predicted ceteris paribus. And the probability of falling into this partition is very high compared to the probability of hitting a long sequence of identical flips.
Maybe it’s because we expect the average coin toss (assigning 1 to heads and 0 to tails) to come out near 0.5, but if all heads or all tails occurs the average is as far away from 0.5 as possible. Then again, a H-T-H-T… alternating pattern would be equally surprising, and it’s average is exactly 0.5, so I don’t know.
There’s an old, old “theory of probability for the layman” book by Warren Weaver which has some material on exactly this topic of “surprising events” vs. “rare events”. For example, any particular bridge hand you are dealt has the same probability as any other, but nobody is going to say “Sorry for doing this, but this hand is so surprising I just have to lay it down and show you - I’ve been dealt the ace, jack, seven of clubs, four small hearts, the queen, five, three of spades, and the king, ten, eight of diamonds”.
“Lady Luck”, first published in 1963, last revised in the 1980s. The examples are perhaps a bit dated for modern readers, but it’s still a good casual read for an introduction to probability:
You are confusing “sequences” and “outcomes”. In short, people don’t care about the sequence, and it is only in the case that you mentioned (99 heads or 99 tails) that the outcome has the same probability as the sequence. People are surprised by the outcome (99 heads) because most other sequences can be aggregated into outcomes like (50 heads/49 tails) that are far more probable than 99 heads.
So if you care about “sequences”, your line of thought is precisely right, they’re all equally probable. If you care about outcomes, then the 99 heads outcome is extremely rare and surprising. The other people you are trying to relate to care about the outcomes while you are focusing on the sequence, hence the cognitive dissonance.
Hope that helps!
The thermodynamic concepts of microstate and macrostate (Microstate (statistical mechanics) - Wikipedia) might be useful to consider. In this case, the sequence observed corresponds to the “microstate,” while the total number of heads corresponds to the “macrostate.” Each microstate is equally probably, but some macrostates (like having 49 heads and 51 tails) are far more likely than other macrostates (like 99 heads and one tail).
What is the probability that something improbable will happen?
It’s surprising because humans are good at recognizing patterns, and all heads is an obvious pattern. And further because the pattern is at odds with our current model of the world.
The OP states that the coin is fair, but the pattern suggests strongly that it isn’t. And in the real world, of course, we aren’t ever sure that a coin is fair, we simply have information from various observations. It’s important to be able to recognize new information like that and feed it back into our mental model.
Sure, it’s possible that a fair coin will flip all heads, but it’s exceedingly unlikely in a way that a mixture of heads and tails isn’t. The “surprise” is our brain going “hold on a second, something isn’t right here…”
There’s a great quote from Richard Feynman: “You know, the most amazing thing happened to me tonight. I was coming here, on the way to the lecture, and I came in through the parking lot. And you won’t believe what happened. I saw a car with the license plate ARW 357. Can you imagine? Of all the millions of license plates in the state, what was the chance that I would see that particular one tonight? Amazing!”
I’ll add one thing here. I think we have some kind of intuitive grasp of what independent and identically distributed means. In the contexts of coin flips, identically distributed means that each flip has the same chance of being a heads or tails. This is true even for double-headed or weighted coins. Independent means that the outcome on one flip doesn’t affect that on the next (or any later one). Both of those properties probably seem intuitively obvious for a fair coin.
If you see: HTHTHTHTHT… you are likely to conclude that the flips are not independent and that the coin is not fair.
If you see 40 heads in the first 50 tosses and only 10 heads in the second 50 but in apparently random patterns, you are likely to conclude that either their is a complicated dependence or that the flips were not identically distributed. Somehow the flipper changed from one biased coin to another or some other trickery was involved.
If there is nothing obviously strange about the sequence, you will likely conclude the coin is fair. But there is no simple intuitive way to define “obviously strange”.
The essence, as Zombiewolf and FasterthanMeerkats have pointed out, is the problem written into the OP’s question: “But of course every single possible sequence of flips has exactly the same probability.” That’s true, but the probability of getting 99 heads - a very specific arrangement - is waaaaaay lower than the probability of getting about 50. And that’s what you’re really comparing.
I want to directly address the main concern of the OP, namely, of the 2^99 different outcomes do we consider all heads (or all tails) special from the rest.
Information Theory!
And since I’m a CS guy, I’m going to link to Algorithmic Information Theory since I know that stuff pretty well.
Think of an out come as a string of characters: TTHHHTHHTTTH …
Some strings look random, some don’t. In particular all heads and all tails aren’t random looking at all.
Okay, what is “random looking”. In Algorithmic Information Theory we can (loosely) define a string as random if there isn’t a program that produces the string that’s shorter than the string itself. Note that we can write very tiny programs that crank out billions and billions of all H’s in a row. So those strings are quite non-random.
Note that only a fraction of all strings can have a slightly shorter program that generates them. I.e., most strings are fully random. And even then, virtually all strings are quite close to random. Only a tiny, tiny number are noticeably non-random.
(Note that there are several “gotchas” that appear if you don’t define everything right. E.g., you have to avoid Berry’s Paradox.)
42
I just thought I’d make a note about this comment because I think people underestimate the number of “non-random” results from these things. The truth is that human brains are great at extracting patterns from noise, even when the pattern is not meaningful.
As an example, last night, I was e-filing tax returns. The previous day’s returns were approved and I thought “How odd: 3 returns, and the last names start with A, B and C. What are the odds?” Then I picked up the new ones to file and thought “How odd: 4 returns and 2 last names start with D and 2 with H. What are the odds?”
As in the OP’s question, the odds of either exact occurrence are extremely low. However, the odds of my brain creating a pattern out of any 3 or 4 tax returns? Pretty good, really, when you consider all of the pairs, sequences and other patterns I could use to create something that seems non-random. The ABC pattern would also be there with BCD, CDE, etc. Or I’d probably see an ACE, BDF pattern as also significant (skipping letters) or AEI (all vowels) or CGO (all rounded). Then there are all the three- and four-letter words that might be created.
Psychics and other con artists can use this propensity to find patterns in randomness. They know that there’s actually a pretty good chance that we’ll find something meaningful even when presented with random gibberish, and then we’ll mistakenly think the odds of such a thing happening are too low to be random.
So an important element for the OP’s question in particular is not just “did you 99 results in a row?” but “were you specifically testing for 99 in a row, or were you just looking for any pattern?” The odds on the two questions are very, very different.
That’s pretty much my answer to the OP’s question. (Or at least one of the answers.) A sequence of 99 heads or 99 tails looks surprising partly because one would expect a fair coin to average somewhere around 50 heads and tails, but partly also because it’s fulfilling a pattern. I suspect 100 flips that alternate heads and tails, despite equaling exactly 50 heads and 50 tails, would be interpreted as equally surprising, because of the pattern involved. Or a sequence of 10 heads, followed by 10 tails, followed by 10 heads, followed by 10 tails, etc.
I do.
Stranger
To say essentially what other people are saying in a somewhat rephrased form, we judge the “surprisingness” of a result not in terms of the actual mathematical probability of that specific result out of the universe of possible results, but rather, in terms of the degree to which it seems “typical” of the process that produced it. That is why, out of 10 coin flips, a result like HHTHTTHTHT does not feel “surprising”, while results like HHHHTHHHHH or HHHHHTTTTT do, even though the probability of each of those three specific sequences is identical.
I wouldn’t find it surprising; I’d find it suspicious. As in, I’d suspect that this supposed fair coin is perhaps not so fair. Or even more likely (since even unfair coins will still come up tails occasionally), the person tossing the coin was not doing so in a fair manner.