More Regression to the Mean

I wrote about regression to the mean earlier this week:
http://jaywacker.blogspot.com/2011/08/regression-to-mean.html
This is what happens when you have a strange result, for instance flipping a coin 6 times and getting all heads, but if the coin is fair, you'll find that eventually, you'll get back to a 50:50 heads to tails ratio.

In particle physics, we have to read tea leaves a lot.  It can take a long time for a result to regress to the mean, in some cases it can be a decade wait.  So it's useful to understand how a result regresses back to the mean.

So let's consider trying to measure if a coin is fair or not.  To do this, we need to have a mathematical model of an unfair coin.    My unfair coin model is defined in terms of one parameter, r, that ranges from -1 to +1.   It describes the probability that a coin will end up heads or tails.  The probability for for a coin to land heads is

• pH = (1+r)/2

while the probability for tails is

• pT = (1-r)/2.

If r=1, then the coin will always end up heads and if r=-1 it will always end up tails.
The binomial expansion is the easiest way to compute the probability of getting M heads in N flips.  We just multiply out:

• P(N) = ( pH Heads + pT Tails)^N

and grab the coefficient in front of the  Heads^M Tails^(N-M) term which is:

• P(M;N) = (N choose M)  pH^M pT^(N-M)

Now comes the fun part where we can do some science.  If we get a result, say 6 Heads and 0 Tails, we can say, our coin had an unfairness parameter, r, would I get this many heads (or more) greater than 5% of the time. The range of r where this state meant is true is known as the 95% Confidence Limit.

With r=0, we would find this result would only occur 1.5% of the time, therefore, we say "that at the 95% confidence level, we have excluded r=0."   In fact, we'd say 98.5% confidence level, but 95% confidence level is a standard ruler.

Okay, so now we can say, what is the minimum value for r that is consistent with 6 Heads and 0 Tails at the 95% confidence level. And the answer is r=0.21.  We can reverse the requirement and say, what is the maximum r so that we would get 6 or less Heads 95% of the time.  Well that is easy, r=1 the maximum value, and r=1 is certainly consistent with getting 6 Heads in a row.  So we can plot the 95% confidence interval below from 0.21 < r < 1.0 :

Of course, if it was a fair coin, 1.5% of the time, we would get 6 Heads and 0 Tails -- not that rare if we flip a lot of coins. So we'd want to keep on flipping and watch how the number of Heads and Tails evolve.  So let's flip the coin another 6 times and then flip it another 12 times after that.  So we'll double the data and then double it again.  The following diagram illustrates the regression

So the green band corresponds to the 95% confidence level on r for the initial measurement, the blue bands correspond to the 95% confidence level for some selected outcomes of the next 6 flips (added together with the first 6) and the red bands correspond to some selected measurements of the following 12 flips (added together with the first 12 flips).    The thickness of the arrows illustrate how probable an r=0 (i.e. a fair coin) will take each path.  The thicker the line, the more probable.  The percentages are shown in the inset box.  The red shows the most probable trajectory for an r=0.0 coin while the blue illustrate the trajectory for an r=0.62 coin.

The thing to notice is that when we double the data, we don't get back immediately to the mean, but that it slowly begins to back off the exceptional result.   So what we see is that we had excluded r=0 at the 95% confidence level with the first measurement, but that the most probable outcome of the next measurement will be consistent with r=0.0 and further measurements make the most probable value for r smaller and smaller. Obviously it's possible to fluctuate up again, but it's not common (as common as downward fluctuations).

In contrast, lets look at an r=0.62 unfair coin.

What we see here is that the 6 Heads and 0 Tails was consistent with r=0.62, in fact it occurs 29% of the time.       Now as we add more data, we accumulate higher values of heads and tails and the most probable values of heads and tails are more and more inconsistent with r=0.    The regression to the mean of r=0 is incredibly unlikely.  

So what does this have to do with physics?  Well, we are trying to discover r!=0 and so we look for exceptional results.   We frequently get measurements that occur only 1.5% of the time.  If you do 1000 measurements, you expect 15 such results by chance alone even if r=0 exactly.   So what's important after an anomaly is seen the next measurement.  If you see a regression towards the mean of r=0, then the original anomaly was probably just a fluke.  On the other hand, if the anomaly keeps on growing, then we're in business!   

With the Higgs anomaly at EPS, we saw something akin to 6 Heads and 0 Tails.    At Lepton Photon we saw that anomaly develop to 9 Heads and 3 Tails.   Now that occurs 7.1% of the time in an r=0.62 coin, but occurs 21% of the time in r=0 coins.    So we can release our breath and take a more wait-and-see approach.  If on the other hand, that anomaly had developed to a 12 Heads and 0 Tails (or even 10 heads and 2 Tails), then we'd be in a different situation.