There has been tremendous confusion here and at Uncommon Descent about what I’ll call the ‘all-heads paradox’.
The paradox, briefly stated:
If you flip an apparently fair coin 500 times and get all heads, you immediately become suspicious. On the other hand, if you flip an apparently fair coin 500 times and get a random-looking sequence, you don’t become suspicious. The probability of getting all heads is identical to the probability of getting that random-looking sequence, so why are you suspicious in one case but not the other?
In this post I explain how I resolve the paradox. Lizzie makes a similar argument in her post Getting from Fisher to Bayes, but there are some differences, so keep reading.
Early in the debate, I introduced a thought experiment involving Social Security numbers (link, link). Imagine you take an apparently fair ten-sided die labeled with the digits 0-9. You roll it nine times and record the number you get, digit by digit. It matches your Social Security number. Are you surprised and suspicious? You bet.
As I put it then:
My point is that for anyone to roll their SSN (or another personally significant 9-digit number) is highly unlikely, because the number of personally significant 9-digit numbers is low for each of us while the space of possible 9-digit numbers is large.
If someone sits down and rolls their SSN, then either:
1. The die was fair, the rolls were random, and they just got lucky. It’s pure coincidence.
2. Something else is going on.
Just as my own SSN is significant to me, 500 heads in a row are significant to almost everybody. So when we express suprise at rolling our SSN or getting 500 heads in a row, what we are really saying is “It’s extremely unlikely that this random number (or head/tail sequence) would just happen to match one of the few that are personally significant to me, instead of being one of the enormous number that have no special significance. Hmmm, maybe there’s something else going on here.”
That helps, but it doesn’t seem to completely resolve the paradox. It still seems arbitrary and subjective to divide the 9-digit numbers into two categories, “significant to me” and “not significant to me”. Why am I suspicious when my own SSN comes up, but not when the SSN of Delbert Stevens of Osceola, Arkansas comes up? His SSN is just as unlikely as mine. Why doesn’t it make me just as suspicious?
In fact, there are millions of different ways to carve up the 9-digit numbers into two sets, one huge and one tiny. Should we always be surprised when we get a number that belongs to a tiny set? No, because every number belongs to some tiny set, properly defined. So when I’m surprised to get my own SSN, it can’t be merely because my SSN belongs to a tiny set. Every number does.
The answer, I think, is this: when we roll Delbert’s SSN, we don’t actually conclude that the die was fair and that the rolls were random. For all we know, we could roll the die again and get Delbert’s number a second time. The outcome might be rigged to always give Delbert’s number.
What we really conclude when we roll Delbert’s number is that we have no way of determining whether the outcome was rigged. In a one-off experiment, there is no way for us to tell the difference between getting Delbert’s (or any other random person’s) SSN by chance versus by design or some other causal mechanism. On the other hand, rolling our own SSN does give us a reason to be suspicious, precisely because our SSN already belongs to the tiny set of “numbers that are meaningful to me”.
In other words, when we roll our own SSN, we rightly think “seems non-random”. When we roll someone else’s SSN, we think (or should think) “no particular reason to think this was non-random, though it might be.”
The same reasoning applies to the original “all-heads paradox”.
Flint,
That’s right. It’s similar to the P(T|H) problem when computing CSI.
Knowing how easily humans are fooled by magic tricks, I would regard that as the most likely explanation for the observation.
However, the probability that it was due to pure luck is still nonzero.
I think all you have to have is a target. If I specify some arbitrary sequence of heads and tails, and then just happen to flip that sequence, this would of course cause me to look for some possible cause besides pure chance. But until that phenomenally unlikely event occurs, I wouldn’t bother hypothesizing a cause. Why bother?
I know the numbers folks in Las Vegas can look at the day’s handle, and tell you with amazing precision what their take should be. So the numbers don’t have to wander outside this range more than a percent or so, before cheating becomes the more likely explanation. HOW the cheating was done might take some digging.
Aargh!
I wish philosophers would stop using this stupid argument.
How about an alternative: “A single observation of a lion refutes the hypothesis that all leopards have spots.” Nobody goes for that one, perhaps because it is obviously stupid.
Captain Cook, or whoever it was supposed to be, did not see a black swan. He saw a black waterfowl of heretofore unknown species and genus. It was not a black swan until somebody decided to assign that name to those birds. The lion is more closely related to the leopard than the black swan is to the European white swans.
Neil,
As a mathematician, surely you understand that the criteria for ‘swan’ or ‘non-swan’ are irrelevant to the truth of my statement, as long as you don’t define swans as white (or black):
A single observation of a fliggish darp can falsify the hypothesis that all darps are non-fliggish.
“Rolling snake-eyes” is a specification which has exactly one roll in it: 1-1.
“Rolling 7” is a specification which has exactly six rolls in it: 1-6, 2-5, 3-4, 4-3, 5-2, and 6-1. For any individual one of the six “rolling 7” rolls to come up, yes, the likelihood of rolling snake-eyes is exactly equal to the likelihood of rolling that one specific member of the six-member set of ‘rolls that come up 7.
Please keep in mind the difference between “the probability of one specific die-roll coming up”, on the one hand, and “the probability of any die-roll which fits a particular specification coming up”, on the other hand. In most cases, these two probabilities are not equal.
Only in an imaginary world where natural languages are strictly logical systems.
I grew up in Australia. In fact, it was near the Swan River. We always called those birds “black swans” as a two word noun. Nobody there ever thought that a black swan was a swan, with “black” as just a descriptive term. We were proud of our distinctive native fauna.
I don’t think I quite understand your objection (if you are in fact objecting to something). My point was that a distribution is different from a sequence, though perhaps like Mike I should have said a combination is not a permutation.
Neil,
Natural languages don’t have to be “strictly logical systems” in order for us to discuss the validity of logical inferences.
Ask 100 intelligent people to answer the following question, and let me know what you find:
I predict that the overwhelming majority will say “yes”, and they’re correct.
I think this is a case of deliberately missing the point. Universal statements are disproved by single exceptions. If you are trying to argue that exceptions to universal statements are always members of some set other than that intended by the universal statement, that’s going to take some work.
For example, take the statement “all integers are evenly divisible by two.” Are you willing to argue that odd integers are not integers at all, but we’ve been tricked into thinking so because of semantic ambiguities?
Over at UD, Eric Anderson has a post that exemplifies the kind of woo that often surrounds probability and patterns.
This is a particularly good question when one considers the system itself, and not one’s subjective reactions about what is “unusual”.
From a physical system perspective, what’s happening in the coin flipping test?
The first coin flips, it’s heads. Second flip heads, third flip too. In a short while, the test stands at 30 flips, all heads. Now, setting aside what a human observer thinks of this, what does the physical system think of this?
Well, duh, it a physical system, impersonal and unthinking. It can’t “consider” anything, of course. While an obvious point, it’s an often overlooked one. The system doesn’t “consider” and so “all heads” has no impact on the system. There is no “reaction of amazement”, no change in the physical processes at all.
It’s just coins being flipped, and physics governing the tumbling and resting position of the coin.
If that’s the case, and that is the case, then what is “unusual” to the system? Nothing, as above, it’s an incoherent proposition. And yet, it is the physical system that governs the outcome of the coin flips, and our “amazement” or “suspicion” or “incredulity” doesn’t have any impact on the system at all.
In that frame of reference then, no, “all heads” isn’t special, because the system that determines the outcome doesn’t do “special” or “unique”, like humans do.
It’s clearer in reverse: a human observer finds “all heads” quite extraordinary and remarkable, and in human terms, it is a remarkable configuration. But with a fair coin being flipped, the human observers are just that — observers. They don’t have any say or impact on what the physical system does. They are ciphers in this context, machines that provide a launching mechanism for the flip.
So if I think of all the answers I might come up with to Eric’s questions — and it’s not hard to explain why “all heads” is noteworthy to me as a person considering outcomes — I wonder what he, or anyone thinks could be the impact of such realizations. If Eric is just watching the coins being flipped, does he suppose that “all heads”, because he somehow finds that interesting in his brain, can affect the physical system with his interest or incredulity. I can say the same thing about keiths’ SSN example. The number generator doesn’t give a damn about whether the outcome matches his SSN. From the system’s point of view, a perfect match to keiths’ SSN isn’t the least bit unusual, noteworthy, or interesting.
Why does the deer that just wandered through my back yard skitter of when it hears just the faintest sound coming from the house? Because paranoia is a good trade. Psychologically, it’s better to spend some extra energy running off “just in case”, responding to lots of false alarms, than to save that energy and be wrong, just once, and end up being lunch for a predator.
Humans have a similar disposition toward designs and schemes. There are low costs for “design delusions” and conspiracy theories that aren’t grounded in real threats or schemes. On the other side, failure to pick up on designs or schemes can be high cost — deadly, even.
So we lean toward error on the conspiracy/design side. It’s a good trade. Falsehoods that way are a “good deal” for humans. But it’s a bias toward error, even if there are solid reasons for it. From a subject-independent perspective, “all heads” *isn’t* special. At all. And the system that doesn’t consider, that can’t contemplate outcomes (other than through us, as parts of that natural system) just does its thing. Twenty heads in a row, already? Doesn’t matter, and has perfectly no bearing on the next flip of the coin. It’s 50/50 in any case, no matter what the previous n flips of that fair coin were.
keiths,
I guess it is because I have lived long enough in the natural world that I would not take such a hypothesis about things in the natural world seriously. You are apparently talking about hypothetical “philosophical swans.”
Yes, you thought you knew about the color distribution of swans; now you have a potential counter example. Did you really believe before you saw this particular swan that all swans were white? Where did you learn that; folklore? Culture? By definition? Are you going to stop the experiment there?
You thought you knew about the distribution of numbers from a 10-sided die and you threw a number that “surprised” you. Did you really believe before you threw the die that your SSN would not come up? Where did you learn that; folklore? Culture? By definition? Are you going to stop the experiment there?
You learn and test and learn as you go. If this is a paradox to you; well, it is a paradox to you.
I suppose that incomplete knowledge can seem like a “paradox” to some; but viewed as incomplete knowledge, it is really not a paradox.
This is the way the world works; and its fun for the scientist. It provides the motivation to go out and observe and test. We don’t stop at one experiment. Scientists are constantly questioning the “obvious.” We learned long ago that dogmatic assertions of “we know it all” are silly (at least some of society has learned that).
I’m not trying to be obtuse or difficult here. These are freshman philosophy exercises. Perhaps they are good for development and training; but is that where we are in this discussion?
Where is this going? Is Ken Hammian “philosophy” trembling in the background somewhere ready to pounce?
Sal made an assertion about the origins of the molecules of life. We have a great deal of knowledge about atoms and molecules and the conditions under which they form complex structures. Do atoms and molecules not form such complex structures naturally? Where did Sal learn that; folklore? Culture? By definition?
We aren’t stopping the experiments because of assertions that “we weren’t there” or that coin flips “prove otherwise.”
eigenstate,
I resent that. Systems find my SSN to be very interesting, unlike yours. 😡
In seriousness, your statement is correct only if you assume that the number generator is fair. But if I see it produce my SSN, I start to doubt that it’s fair and to consider the possibility that it does “care” about my SSN.
It’s not just a cognitive bias. There is a genuine reason to be suspicious, even though my SSN is no more or less probable than any other 9-digit number under the assumption of fairness.
The reason to be suspicious is that the fairness assumption itself might be wrong. Someone (or some causal factor) unknown to me may actually be causing the number I roll to match my SSN.
I am comparing two hypotheses:
1) I rolled my SSN, and it’s just (literally) a 1-in-a-billion coincidence, or
2) something other than pure chance caused my SSN to show up.
#2 is more plausible than #1. The only reason to prefer #1 would be if one were so absolutely positively certain that the die and the rolling process were fair that hypothesis #2 could be eliminated.
As I said in the OP:
Maybe you are missing the point.
My point was only that the alleged black swan case is a terrible example, because it doesn’t actually fit.
Neil,
You’re expending a lot of energy to persuade us that black swans aren’t swans. Is it worth it?
Have you considered the possibility that you are teaching us something about prickly Western Australians, instead? 🙂
My point would be that Las Vegas owners know many ways to cheat and many ways that gifted people can turn a game of chance into a game of skill. Not following chance is apparently against the rules.
Stage magic is another arena for testing beliefs about cause and effect. One doesn’t have to know how an illusion is done to believe it is a trick. Many tricks have been revealed, and most professionals take pride in revealing their illusions are based on skill and cunning rather than magic.
Which is why most of us think that natural causes are a more likely explanation of biological phenomena. Nature is subtle, but not capricious.
Sure. As would I. The assumption of fairness is easy. Demonstration and calibration of fairness is exceedingly difficult. It’s general “what’s left” when you exhaust available schemes for pattern, purpose or plan.
There’s a transcendental problem, here, no matter what “design bias” we may have. There’s no way to transcend the system we are experiencing to judge it randomness, it’s fairness. We can’t judge what the likelihood what we have seen is a “fair fluke”. Estimating probabilities and plausibilities for non-random hypotheses is the easy part. As one of my kids asked me, recently, but if it was just a product of random outcomes, how could you *show* that?
Got that. I think I was driving at either a restatement of your paradox, or a corollary to it. If we are analyzing competing hypotheses for explaining what we see, randomness is intractable. We can try to gauge ideas about design, cheating, other impersonal but cumulative or otherwise biasing processes, etc. But at the end of the day, you don’t have any basis for judging the fairness of the coin flips on its own merits. It can only be contrasted with other competing hypotheses.
Right. The problematic is issue is judging 1). Stipulating that your SSN is a one-in-a-billion roll of the dice, what are the odds that you will “beat the odds”. If that’s not clearly problematic, let me point to an example with the digits of pi.
It just so happens that in calculating out the digits of pi, the digit-string “999999” occurs at position 726 — the so-called “Feyman Point”, as math geeks would know it. Now, the digits of pi have been calculated out into the trillions of digits, and based on that, the digits of pi appear to be completely random.
So, given that, what are the odds of finding a string of 6 digits in a row occurring within the first 1,000 digits of pi? The odds are tiny. Six 9s in a row happens before any other series of 4 or 5 of the same digit, it turns out.
Should I be suspicious of pi being somehow structured or non-random in its digit sequence from that? If not, why not?
The gist of this should be clear. If we calculated how soon in looking at the digits of pi we should encounter six 9s or six of any digit in a row, it would far, far past position 762 on average. So, in this case, we — er, reality? — “beat the odds”, and we have a “fluke” in finding 6 9s in a row occurring in the first thousand digits, something we would expect to see less than 1 time in a thousand trials against a random source.
I think that’s apropos to your SSN doubting (which, again, I would share and entertain just as you would, if it happened). We have warrant to be suspicious, but we don’t have a way to rule out, or discount as a plausible path, the fluke, the one-in-a-million hit that happened on the first pull of the one-armed bandit.
I think that is overstated. There cannot be any “absolutely positively certain”. See my example with the digits of pi. How could you possibly be certain that the Feynman point, 6 9s in a row at position 762, is just a fluke?
I cannot see any way to even approach the kind of certainty you ar requiring.
And if that’s not available, then what? #2 wins every time, no matter what really obtains concerning #1? I think that is much closer to what really happens with humans than we like to admit. If your SSN is a one-in-a-billion chance, then even if your best competing theory is a one-in-a-trillion chance, #1 still can’t win, because the “meta-hypothesis” lurking there is “there’s an unkown, undiscovered answer I haven’t thought of yet” in #2, and the odds of that obtaining intuitively get pegged at a much higher probability than a billion-to-one.
This means there’s a practical floor on the probabilities of #1. Below a certain level of improbability, the “unknown hypothesis” — the competing idea that *something* non-random is going on, even if I have no clue what it is, trumps randomness and raw probability. That’s the way we are wired. But even if that’s a practical limit, it’s one imposed by our psychology, not because one-in-a-billion events don’t happen, and happen all the time.
Mike,
Hypotheses are held with varying degrees of certainty. (C’mon, Mike, you’re a scientist. Why are you making me tell you this?) Before the discovery of black swans, I suppose Europeans thought that all swans were probably white. Generations had come and gone without seeing a non-white swan. It doesn’t mean that they thought black swans were impossible, just that they thought all swans were probably white, based on past observation. You do the same thing every morning when you flush the toilet and expect the water to flow down, not up.
I thought it would probably not come up, with odds of 999,999,999 to 1. I had no evidence that the die was asymmetric in shape or weight, or that someone was manipulating the throws to rig the outcome, so the fairness assumption was a good provisional hypothesis.
Not if I have a choice, but you’re missing the point. It’s not that I want to stop the experiment there. Suppose (for whatever reason) that I have to stop the experiment, against my will. Is all lost? Have I learned nothing from that single observation? According to you, I’ve learned nothing:
Maybe you don’t, but I do:
Mike:
Who said that was a paradox? Mike, could you please rein in your imagination a bit? Try to limit yourself to arguing against what I’ve actually written.
The paradox comes from difficulties in interpreting the knowledge we have. We flip the coin and get all heads. How do we interpret that information?
Where did you get the strange idea that I advocate “stopping at one experiment”?
Has anyone here claimed that “we know it all”? Who are you arguing against?
No. We are trying to resolve a paradox that has confused a lot of smart people, including you.
Yes, Mike. ‘Keiths’ is just my nom de guerre. I’m really a creationist trying to screw with your mind.
Not even Sal is arguing that we should “stop the experiments”. Geez, Mike.
Mike,
Communication via blog requires using words.
But don’t make this harder than it needs to be. I am not using those words in an idiosyncratic or sloppy way.
keiths,
I can’t identify with this way of thinking about the question.
Eigenstate seems to agree with you that this may be a “wired-in” psychological tendency of humans.
I would question whether it is some innate tendency we have or whether it is cultural; because as I think back over rare, one-off events that I have encountered, if I ever had reason to think something other than pure chance was operating, I think I can trace it to cultural influences. If there is no way to follow up and check, then it simply remains unanswered.
It appears that if it is and innate tendency, it gets trained out of an individual by experience dealing with the art and science of capturing and vetting data.
My automatic response to such things is to immediately run the experiment again; not to think that something is necessarily wrong. My immediate thinking is to see such things as an opportunity to check if things are working. I don’t decide on “skullduggery” until is see more results. If I learned that perspective along the way somewhere, I have no recollection when that actually occurred.
It might be connected with “personality types” or cultural and childhood influences one absorbs from that attitudes of people who are close. Maybe it has something to do with being a “romantic” or being a “hard-headed realist;” and I am not attributing any value to either of those personality traits since I apparently have some of those characteristics. In fact, I suspect most people are not pure “personality types” of any kind.
Curiosity seems common in many animals, including humans. We want to check things out with whatever sensors we have. This seems to be related to the “sense of wonder” in humans. But does that necessarily translate into “suspicion” that something is not normal?
What about adaptability? Adaptability requires becoming familiar with changes and adjusting behavior in response. Closed-in fear and suspicion seem to be connected with reluctance to “break routine.” Early traumatic experiences quickly become associated with pain and fear, which in turn limit curiosity and exploration.
We see these things in many animals. Animals raised from birth in rich, stimulating environments learn and adapt more easily. Those that are raised from birth in limited environments and frequently encountering “punishment” for exploration don’t explore and learn as easily.
So I question whether this tendency of suspicion is “innate;” I think it has a lot to do with very early experiences. With humans, culture appears to have a rather large influence.
Mike,
I think he and I agree that we (and many animals) have a tendency to “be paranoid”, loosely speaking, since erring on the side of caution is a better evolutionary strategy than being cavalier. But my impression is that we still disagree on whether someone is objectively justified in being suspicious when he flips 500 heads using an apparently fair coin, or rolls his SSN number with the ten-sided die.
I’m going to reread his latest comments after I’ve had some sleep.
Mine too; I am an engineer who has spent much of my career debugging complex digital systems in simulation and in the lab. You don’t survive very long in that environment without being extremely careful about confirming results and not jumping to conclusions.
Again, you seem to think that I advocate jumping to conclusions and “stopping the experiments.” I don’t.
What I’ve been saying, and will say again, is that you can get useful information from one-off experiments. That doesn’t mean that you should stop the experiments after one observation. It doesn’t mean that you should jump to unwarranted conclusions. It does mean that even a single observation can yield useful information.
Again:
Mike:
I don’t mean ‘suspicion’ in a negative sense. I just mean that we suspect that our fairness hypothesis may be incorrect. It isn’t necessarily bad news if the fairness hypothesis is wrong.
I think you’re wrong about that. For example, if you hear something rushing toward you in the dark, your immediate response (if your amygdala is functioning normally) is fear. You don’t decide that “Oh, everything’s fine, I’ll bet it’s just a loved one rushing to give me a hug.”
I would say suspicion is warranted upon seeing just a dozen heads in row. Seeing 500 heads would objectively be the basis for a strong sense of incredulity. I won’t presume your position, but I’d be surprised if we were disagreed on that. The problem I was pointing at was not that we do not have objective grounds for suspicion in such a case — we do. Rather, we don’t have any basis for a positive affirmation of a “fluke”. You can’t “show it’s random”, but can only “fail to show it’s non-random”.
I think the only useful information you get from one-off experiments is the inclination to form an hypothesis and test it.
Even in fields like prediction of earthquakes, the critical thing is not the prediction itself, but the analysis of causation.
In Dembski’s analysis of election irregularities, he presents an hypothesis of causation and a means of testing it. Oddly enough, I saw this same election analysis in my college student government elections nearly 50 years ago.
petrushka,
You don’t think I’m justified in changing my belief from
to
…on the basis of a one-off experiment in which I flip 500 heads in a row?
I thought we were discussing 9 tosses, not 500.
At some point we become suspicious.
But I keep trying to point ot that coin tosses have no relevance to the specific biological issue that started this discussion.
petrushka,
But that contradicts this:
I would argue that 500 coin tosses is not a one-off experiment. After a dozen or so heads you form the hypothesis, and the remainder are the test.
Of course the experiment could be double blind.
But we do not go into such experiments without expectations. We expect a distribution.
I actually come across this as a practical problem in my daily work. Often, with cognitive psychology, we want to present stimuli “randomly”, for each person. But it creates problem, because, after enough participants, you are bound to find some occasion when a participant ended up never being presented with one class of stimulus, because they got the equivalent of “all heads”.
One solution is to do “random without replacement” – so we have a set ratio of heads to tails (metaphorically speaking), but a randomised draw (like drawing red and blue balls from a bag, without putting them back each time).
But that’s not ideal, because some smart cookies will figure: “hey, I’ve just had a lot of heads – that means the next lot will be mostly tails”.
That’s why it’s really misleading to say that some sequences are “random” and some “non-random”. The adjective “random” should refer to the generation process, not the sequence itself. In fact, what we do sometimes is to present “pseudo random” sequences, in which we pick a few randomly generated sequences that “look random” (in other words we non-randomly pick them!) and use those.
And that raises the issue of how you randomly generate anything. There are very few natural processes that give you a series of outcomes where each is completely independent of what has gone before (total absence of autocorrelation).
So far from “natural” processes being “random” or “chance” processes, natural processes are annoyingly “design” like!
But the entire discussion has been premised on the notion that flipping a coin 500 times, or rolling a 9-digit number, is the experiment!
eigenstate,
Yes, if you’re talking about absolute certainty. Not if you’re talking about a high probability. After all, the probability that something is a “fair fluke” is just one minus the probability that it is not fair. If we weren’t able to determine the latter, we’d have no objective basis for our suspicion when flipping 500 heads in a row.
I used to joke that I could sell a pseudorandom number generator routine that did nothing but
When customers complained that it was nonrandom, I would point out that every truly random number generator will eventually produce arbitrarily long sequences of all ones. The customers would never be able to prove that my routine was nonrandom (unless they disassembled the code). But getting a few hundred billion results of 0xffffffff would certainly justify their suspicion.
One in a billion. “Beating the odds” just means that I get my SSN, and that happens once every billion trials, on average.
That’s an interesting problem. There are analogous problems in computer science where you want a sequence that’s “kind of random, but with really long streaks filtered out”.
I’ve had a lot of experience with random number generation; the need for it comes up frequently in signal and image processing. These days, one can get “off-the-shelf” random number generators; but you still need to test them when you get them. Early in my career, people designed and tested random number generators on their own; they weren’t readily available commercially.
I have had to design and test random number generators to produce desired distributions for injecting noise of a particular type into signal and imaging processing hardware and software to be able to uncover subtle effects of both the processing and the noise characteristics.
This is but one of hundreds of different issues encountered by anyone who works with pulling signals and images out of noise; so it is not something that one can avoid doing.
There are a number of tests of random number generators that have become fairly standard over the years. You can find them in various multivolume tomes like Knuth.
The spectral test is one of the easiest to set up and vary. One bins the results of the random number generator and enumerates what falls into each bin. Then you vary the bin boundaries to account for errors that occur at binning boundaries, both from the random number generator and from the logic that selects a bin for a given result from the generator.
The variations in the distribution of counts in the bins can then be compared with the ideal mathematical probability distribution you are trying to generate. You can vary the bin widths as well as the boundaries in order to vary the resolution of your test.
One can pick up very subtle differences from an ideal distribution this way. It is both visual when the histogram is plotted and quantitative when compared with the ideal mathematical distribution.
Well, clinical trials are experiments, but there have been cases where the effectiveness was so obvious right away that the trial has been cancelled.
But it is not just the series of heads, but the combination of the series plus a theory of causation.
In the case of coins and dice, we have expectations that gambling devices can be rigged, so we can quickly come to the conclusion that tampering is likely.
Same with electronic pseudo-random generators. We know that software bugs and hardware glitches occur, so any deviation from the expected distributions suggests a problem.
the problem with Sal’s hypothesis (did he have one?) is that magic has never survived a rigorous examination of protocol, so magic is not high on the list of expected causes.
I think we need to distinguish between the unexpectedness of a series and the expectations of causes derived from the history of similar experiments. It is not so much that we find a series odd, but that we form reasonable hypotheses.
If we cannot run the experiment again, we are stuck with precedent.
The usual solution to that problem in physics experiments is to reseed pseudo-random number generators with numbers generated from physical noise sources. But one still has to test the generation process.
In real situations, you adjust the need for “randomness” to the resolution of your signal and imaging processes. There is no need to do any better than the signal processing is capable of detecting.
In dealing with humans, if you ask them to blacken a given number of squares in a grid randomly, you find that what they fill in is not random at all. That is because they will typically avoid filling in adjacent squares in a grid. The perception of randomness doesn’t correspond to true randomness.
petrushka,
I was just disputing your claim that
The experiments we’ve been discussing — flipping a coin 500 times and rolling a ten-sided die nine times — show that your statement isn’t true. If I flip 500 heads, that doesn’t merely allow me to “form a hypothesis”. It changes my belief about what is likely to be true.
That’s very useful.
http://www.defencejobs.gov.au/airforce/engineering/
So I write badly. In my mind, an event that causes you to devise a testable hypothesis has changed your beliefs. Prior to the coin flipping your default belief would be that the coin is “fair.”
After the flipping your default hypothesis is that it isn’t fair.
Exactly in what way it isn’t fair is unknown.
I would argue that in the case of biology, there were originally two coins — one two-headed and one two-tailed. One was picked, and subsequent flips make no difference. That’s the hypothesis.
In some sense, all of science is based on the assumption of regularity, as opposed to capriciousness.
One of the issues is the elapsed time during which one is watching the experiment. If results are coming in slowly enough for you to be comparing and thinking about the accumulating results, it is quite natural to start forming “hypotheses.”
But take away the time to think as results are coming in and the problem becomes far less acute in the mind of the individual. If your random set of numbers appears in effectively a flash, you simply look at the number and run the test again. You don’t start forming hypotheses until you have seen more results.
The spectral test I mentioned in one of my earlier comments reruns the test as many times as you wish and generates results that you can look at visually. One’s “hypotheses” are now guided by far more complete data.
Which all goes to show that equivocation (mostly inadvertent) between various definitions of “random” or “chance” lies at the heart at a lot of the heat. “Random” is often taken to mean “non-intentional”, and that therefore anything “non-designed” is “random”.
You could say it’s an occupational hazard of these discussions.
heh.
I do partly blame Monod, much though I like his book.
Absolute certainty is non-starter with me (I guess I can be absolutely certain I am conscious, but beyond that…). Plain old “certain” wouldn’t obtain, either, IMHO, but not looking to debate the semantics of those terms with you. I think we understand each other.
That’s funny. But if you’ve been working on technical frameworks that rely heavily on random (and ergodic) inputs for as long as I have (and it sounds like you have), you’ve no doubt had the occasion to see the random source produce some very non-random appearing inputs. I think Mike Elzinga touched on this, but humans routinely will pick out less patterns than more random alternatives to avoid clustering. To humans, random is something like “evenly blended, but just avoiding patterns and repetitions”.
lol @ imagining the tech support calls coming in for your product, there. Gonna have to pay over-market for your support staff, giving back some of the savings on your engineering budget…
Yes, of course. I was driving at the human expectation that for a one-in-a-billion chance, unless that pull came up somewhere after or near half a billion trials (the “on average” time to expect such an outcome), that the game was rigged. But the time-to-pull for you SSN occurs on the same bell curve — much more common to see it show up “in the middle”, but “first” is still consistent, just out on the tail. That point risks pedantry, so will let that go, but I often hear resistance to what looks like a fluke, after some solid investigation, dismissed because the (ostensible) fluke happened “early” vs. its mean time to happen on the probabilities. “Early” implies unlikely, etc. and should rouse suspicion, but that concern recapitulates the general kind of incredulity at outcomes from random draws we’ve been talking about above.
Well, not “like” exactly – it’s awfully French. But I do like the concept of teleonomy.
I fail to see what is paradoxical about this really.
“As has been discussed, even when the experiment involves generating a permutation of H’s and T’s, it’s very natural to consider the experiment where we generate a combination of H’s and T’s (just forget the ordering). ”
I gave the C(n,r) formula here:
But there is a nuance in its use. Suppose you had a set of 500 coins and each coin occupies a position (1 to 500) in a row.
The C(n,r) formula works from the presumption you started with 250 coins heads and 250 coins tails, and then you find ways that you can distribute it among 500 unique positions.
The permutation issue arises if you put one of the 250 heads coins in position 1 before you put another in position 2 (or whatever position). Permutation deals with the process of how you place the coins in the row, whereas if you’re concerned only about outcome you use the combination formula as I put forward.
If I’m stating it poorly, a hopefully better explanation is here:
http://www.mathsisfun.com/combinatorics/combinations-permutations.html
Regarding KeithS analysis, in the case of all coins heads, there is no need to appeal to any advanced knowledge except that the advanced knowledge the coin is fair. The calculated expected experimental result will suffice. Making one sequence special in ones eyes prior to the experiment is a sufficient but not necessary condition to reject the chance hypothesis.
All fair coins heads is special in our eyes because it deviates significantly from expectation, it does not deviate from expectation because it is special in our eyes!
Further, all coins heads actually gives us exact sequence information (500 bits), whereas 50% coins heads only give us :
-log2( C(n,r) / 2^500 ) =
-log2( C(500, 250) / 2^500) =
-log2 (1.17 x 10^149 / 2^500 ) = 4.8 bits of information
We have surprisal because 500 coins heads is a highly specific sequence of 500 bits (compared to only 4.8 bits surprisal for the case of 50% heads). And even though such surprisal is a sufficient reason to reject the chance hypothesis, it is not a necessary condition since instead of appealing to surprisal, one can appeal to its enormous deviation from experimental expectation.
With regard to the SSN, surprisal value is key. And I can agree with KeithS there, but I don’t agree that it is necessary, even though it is sufficient in the case of all coins heads.
At issue for the SSN is the probability our psychological pre-conception of an improbable event will perception will actually align with an improbable event.
One can use the same reasoning with all-coins heads, but it is not necessary when an alternate proof that appeals to expectation values and the law of large numbers suffices. But such a purely statistical approach doesn’t nicely fit the case of SSN.
With regard to homochiraltity, we don’t have to appeal to pre-specification when appeals the Law of Large numbers and expectation values deducible from chemistry will suffice. I prefer that because pre-specification leads to needless controversies.
The presumption that LUCA’s ancestors were heterochiral is pure speculation, and further, there is good evidence without homochirality there would not be stable protein folding or for that matter much workable biology. This article argues for the necessity of homochirality over heterochirality:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2203351/
Oddly, they say it had to evolve in order to give it survival advantage without even considering the fact heterochiralty might prevent evolution in the first place since the organism could likely be dead…
PS
1. slight unfairness in the coins would still result in rejecting the chance hypothesis, calculations ought to bear this out.
2. I can’t explain why my fonts are blowing up when I preview my post. They make it look like I’m trying to highlight a point when I’m not. It looks ok however after I post.
Sal, I honestly think you are slightly confused.
Strictly speaking, if we found 500 Heads, with no knowledge of how they were generated, the sequence would NOT have 500 bits, because you also need to take into account the frequency distribution of the observed sequence.
Because we know that coins have two ways of falling, which are supposed to be equiprobable when tossed, if we know they were tossed, then we can infer 500 bits (as all sequences of 500 bits have). But if we don’t know they were tossed, then we have to estimate the probability distribution from the sequence
Even if it was 499 Heads and one Tail, that would give you far fewer than 500 bits, because by the time you’ve seen so many heads, your expectation of seeing more heads is far higher than seeing tails.
This is not a trivial point, especially if people extrapolate to sequences of unknown provenance.
I also think you may be succumbing to math voodoo! The binomial theorem is dead simple – the only reason we expect more even-ratio’d sequences than extreme ratio’d sequences is that there are far more of them! In other words, we base our expectations on what has previously happened. If we see lots of even-ratio’d sequences we will come to expect them, and infer that the process that generated them is a process that generates far more even ratio’d sequences than extreme-ratio’d sequences. But not all processes do (not even “natural” processes).
Have you read my piece on Fisher and Bayes?
It’s pertinent! Causal inferences have to be based on far more than an observed pattern, which is the basic flaw in Dembskian ID (the dominant flavour at UD, and enshrined in the UD FAQ).
That’s why Bayes is important in my view. It’s important, when estimating our confidence in the probability of future events that we have some knowledge of frequencies of similar events in the past. In other words probabilities about one-off events only make sense if we also consider our knowledge.
Which how all the argy bargy started – different people made different assumptions about what we knew.
It’s because the wysiwyg plug in isn’t very good!
Sorry!
“you also need to take into account the frequency distribution of the observed sequence.”
I’m afraid I don’t really understand why since we can take the coins and empirically test them after the observation to determine they are fair.
I have not read your essay on Fisher and Bayes. I should do that.
The frequency distribution might be very important if that’s all the data we have, but we can test the coins and generate gobs of data to determine expected frequency distribution.
As I said, we can work from the presumption the coins are fair or approximately fair because we can examine the coins after the observation in case we have doubts.
That also makes the analysis a lot simpler. A much harder issue is if all we have to go on is the observed pattern, in which case your concern is valid.
Well, exactly – so this is important if we are to extend this hypothetical example to making a design inference from a pattern whose provenance we don’t know. That’s why I keep banging on about how important it is to know the probability distribution under the null!
Yes. As I say, the knowledge you have needs to be factored into your probability calculation.
Strictly speaking, no.
The C(n,r) formula is just a count of the possible ways of getting particular combinations, without regard to the order in which the heads/tails occurred.
A theoretical analysis of a fair coin tells us that either possibility should have a probability of 0.5 for each toss. Combining this with the C(n,r) formula, we can compute the probability of each possible combination. That each possible sequence (where the order is important) has the same probability comes from that theoretical analysis of the fair coin.
Lizzie:
And Sal, just so you don’t miss the message — Lizzie’s criticism applies to your homochirality argument.
This statement from your OP illustrates the problem:
That’s simply wrong. There is no reason, provisional or otherwise, to assume that the binomial distribution applies to homochirality in biology.
Hi Sal,
“As has been discussed, even when the experiment involves generating a permutation of H’s and T’s, it’s very natural to consider the experiment where we generate a combination of H’s and T’s (just forget the ordering). ”
I gave the C(n,r) formula here:
But there is a nuance in its use. Suppose you had a set of 500 coins and each coin occupies a position (1 to 500) in a row.
The C(n,r) formula works from the presumption you started with 250 coins heads and 250 coins tails, and then you find ways that you can distribute it among 500 unique positions.
The permutation issue arises if you put one of the 250 heads coins in position 1 before you put another in position 2 (or whatever position). Permutation deals with the process of how you place the coins in the row, whereas if you’re concerned only about outcome you use the combination formula as I put forward.
I’m not sure if I follow you 100% here, but looking back, maybe I should have referred to sequences of H’s and T’s and multisets of H’s and T’s instead; in any case, if you’re explaining the distinction between permutations and combinations, then I think we must agree.
In the post where I typed that quote, I was just describing how we could view the 500 coin flips as an outcome of two distinct experiments, one where we keep track of the positions of the H’s and T’s in the 500 slots, and one where we don’t (so we just keep track of the total number of H’s/T’s in that case).
In the first experiment, the sample space has size 2^500 and the distribution is uniform, so every single outcome has the same probability. In particular the outcome of 500 heads is exactly as likely as any other ordered sequence. Which illustrates the paradox: Why would we become suspicious if 500 heads came up rather than some random looking sequence of H’s and T’s, when they have the same probability, assuming the coin is fair? (I certainly would be suspicious, in fact I would begin to suspect something is up after the first 6 or 8 heads!)
In the second experiment, the sample space consists of 501 outcomes, definitely not all equally likely, and 500 heads is extremely improbable compared to, say, 250 heads and 250 tails.
Anyway, that’s a bit longer a reply than I intended, but do you pretty much agree?
“Strictly speaking, no.
The C(n,r) formula is just a count of the possible ways of getting particular combinations, without regard to the order in which the heads/tails occurred.
”
I was merely pointing out that if we have 250 coins heads and 250 coins tails, how we are able to determine the number of sequences that accord with that constraint.
The number of exact ordered sequences conforming to the constraint of 50% heads in set of 500 coins is approximately
1.17 x 10^149 or about 3.57% of the possible 2^500 ordered sequences.
It’s confusing because we’re using the combination formula to count the number of permutations (exact sequences), and this seems counter intuitive to do the counting in this manner!
The confusion can be reduced if we visualize 500 coins and each coin will be placed in a numbered bin.
To illustrate with the simplest case, consider we had 500 coins heads in one pile and take coins from the pile and put it in the bins. There are 500! ways (processes) of filling the bins.
Example:
Process 1: We can put one coin in bin 1, then another in bin 2, etc.
Process 2: We can put one coin in bin 500, then another in position 499, etc.
…..
Process 2^500: We put one coin in bin ….
Each process will still result in 500 coins heads, but there are 500! ways of filling the bins.
But it really doesn’t matter that there are so many ways (process permutations) we can fill the bins, what matters is that there is only 1 sequence permutation that is all heads.
But the number of process permutations resulting in all heads is not the same as the number of sequence permutations with all heads. And the confusion arises because it is not always clear which permutations we are talking about: processes or sequences.
But once clarified we can see the appropriateness of using the combination formula (for processes) to count the number of permutations for sequences conforming to all coins heads:
C(n,r) = C(500, 500) = 500! / 500! (500!-500!) = 1
Maybe we could say:
Cproc(n,r) = Num sequences (permutations) with property of r heads
Similarly the number of process permutations resulting in r heads is not the same as the number of sequence permutations with r heads.
Thus the combination formula for the number of processes arriving at 50% heads is actually equal to the number of sequence permutations of that are 50% heads. Thus:
C(n,r) = number of sequence permutations with r heads