Regression to the mean...

Statistics is important in physics.  The primary reason is that quantum mechanics says that you can not predict the outcome of any single experiment, but only the distribution of outcomes.  This sounds very mysterious and bizarre, but it's not that unfamiliar.  Let me describe a toy example of a quantum mechanical theory: a coin flip.

So you can understand everything about your coin, but if you flip the coin high enough, you won't be able to predict whether it will land heads or tails on any given flip.  However, if you study the coin in enough detail, you can determine what fraction of the time it lands heads or tails.  If it's a fair coin, then it will land 50% heads and 50% tails.  I'll use a fancy name -- the fair coin hypothesis -- to describe the assumption that the coin is fair.

So let's say we want to test the fair coin hypothesis.  How would we do it?  Well you'd start to flipping a coin and start logging the results.  Now if it's a fair coin, you can compute the probability that you'll get M heads in N flips of a coin:
 P(M; N) =   (N choose M) / 2^N
where (N choose M) is
M!/(N! (N-M)!).
If you work really hard you can show that this formula gives a normal distribution centered around M = N/2. Cool.  (If you want to prove this yourself, use the Stirling approximation or see: Quora answer)

So how do we use this to test a hypothesis?  Well we can start asking, "What is the probability that we have a fair coin given our results?"  This is a fairly straight forward question, but not quite precise enough.  A better question is, "If we have a fair coin, what is the probability that we get as exceptional of a result as we observed?"   So let's take an example of 6 coin flips.  The probability we get any given result is

• 6 Heads or 0 Heads: 1.56%
• 5 Heads or 1 Heads: 9.38%
• 4 Heads or 2 Heads: 23.4%
• 3 Heads: 31.3%

If we saw 5 Heads and 1 Tail, we would say that a fair coin would give this exceptional of a result  21.9% of the time.  Which is 5 Heads, 1 Heads, 6 Heads, 0 Heads.  So about 1 out of 5 times we would get such this weird of a result... which is pretty common.    Now if we saw 6 Heads and 0 Tails we would say that a fair coin would give this exceptional of a result  3.13% of the time (6 Heads or 0 Heads are as exceptional as this).  This is getting pretty rare.   We might start to get suspicious that the coin isn't fair.  Of course we don't know that it's not fair, this type of result happens occasionally.

There is a theorem in statistics that goes under the name regression to the mean. What this expression says, is that if you get an exceptional results -- say 6 Heads & 0 Tails -- that if your hypothesis is right and you keep on performing the experiment, you'll find you'll get the average result eventually.  

So if we flip the coin another 6 times we'll have the same probability as above, but we need to add the new results to our old results and we get

• 12 Heads: 0.024%
• 11 Heads:  0.26%
• 10 Heads: 1.6%
• 9 Heads:  5.4%
• 8 Heads: 12.1%
• 7 Heads: 19.3%
• 6 Heads: 22.4%

Now given if we get 5 or 6 Heads in the second round, we'd start to be getting pretty suspicious. There is only a 0.5% chance of getting 11 or 12 Heads in 12 flips.   However, if we got  4 Heads, we'd still have a fairly exceptional result of  10 Heads in 12 flips, but 3.7% of the time we'd get that, which means that we doubled our data and the exceptional result didn't get more exceptional.   If we got 3 Heads in the second round we'd be up to a 14.1% chance of getting this result -- the result would be less exceptional.  If we got 0, 1 or 2, we'd be back to the mean.  

If the first result wasn't a fluke, the further results would begin to accumulate around 12 or 11 flips if the coin wasn't fair.

So what does this have to do with the Higgs boson?  Well at the EPS, the experiments at the LHC saw a 1 in 1000 fluke that could be interpreted as a Higgs boson.  A 1 in 1000 fluke sounds weird, but we do a lot of measurements at the LHC, so flukes happen just by chance.    So we usually set the bar for discovery very high: a 1 in a 1,000,000 fluke (in terms of standard deviations, 5 sigma).  The only way to get that is to take more data and see if the result becomes more exceptional.

So today at the Lepton Photon conference in Mumbai,  the experiments added in 60% more data.  If the 1 in 1000 fluke was real, it should begin to become closer to 1 in 10,000 fluke rather than staying a 1 in 1000 fluke or reducing to  1 in 100 fluke.

We saw that the data stayed being a 1 in 1000 fluke.  This is kinda like the example of getting 6 Heads in a row and then in the second batch getting 3 Heads and 3 Tails -- the most typical outcome for a fair coin and the combined 9 Head and 3 Tails isn't that unusual and it makes it seems like the result is regressing to the mean.