Regression to the mean cub. Understanding regression to the mean. Application in the world of finance

Do you believe that after great luck there is always a streak of bad luck? For example, if today you received a really strong deal in poker, then tomorrow even the machine that dispenses shoe covers will ignore you. Or maybe you think that your talent for cutting with a jigsaw or your unearthly beauty must be inherited by your children? If you are sure of this, then statistics speak out more restrainedly on this issue. A statistical principle called “regression to the mean” will help explain such phenomena. Ignoring it can lead to, at a minimum, a bad mood, and at a maximum, to complete disappointment in one’s life. Actually the idea is very simple. Let's sort it out.

Talent or genius, great luck, failure or other extraordinary phenomenon are extremely rare, that is, the likelihood of their occurrence is extremely small. The probability of a repetition of such a rare event will be even less, since probability multiplication is used to find it. Thus, after any extreme event (good or bad), everything returns to normal. There is a very important point here - life does NOT compensate for your failures or victories, it’s just that your luck indicators rush towards their average values. This is regression to the mean (from the Latin regressio - reverse movement). The same thing happens with the change of generations. Your children will definitely be talented, but most likely in a different area.

The concept of regression was first introduced by Sir Francis Galton, an English generalist researcher. He is responsible for another fundamental concept of statistics – correlation. While studying heredity, Galton measured everything that could be measured in his compatriots: heads, noses, hands, the number of fussy movements, the degree of attractiveness, etc. Galton believed that a person’s character, his mental abilities and talent are also determined by heredity and are subject to the principle of normal distribution.

In one of his works, he tried to find a connection between the height of parents and the growth of their children. The dependence is obvious - tall parents give birth to tall children and vice versa. But Galton, in addition to this, also discovered some not entirely logical patterns. For example, he found that parents with above average height had tall children, but they were not as tall as their parents. And parents with below average height had children who were short, but not shorter than their parents. This means that the height of adult children deviates less from the average than the height of their parents. That is, descendants “regress” more strongly to the mean. Actually, Galton called this phenomenon “regression to mediocrity,” which more accurately reflects the meaning, IMHO.

Galton constructed a graph that resembles a modern scatterplot.

He divided people into groups depending on their height (in inches), calculated the arithmetic mean for each group and marked these values ​​on the graph. Next, Galton approximated these points and constructed straight lines, the so-called regression lines. Galton even calculated the correlation coefficient - 2/3. This means that only 67% of children's height is determined by the height of their parents.
The graph reads: “When the average height of parents is greater than the average height of the population, children tend to be shorter than their parents. Conversely, when the average height of parents is less than the population average, children tend to be taller than their parents.”

Although Galton's conclusions and ideas are now mildly questioned rather than criticized, they have revolutionary significance for statistics. Thanks to this versatile scientist, regression and correlation analyzes are now widely used.

Below we have constructed a scatterplot (aka scatterplot) for the data collected by Galton. In 1886, he presented a tablet showing the heights of 928 adult children and the heights of their 205 parents (a weighted average of the heights of the father and mother). Since then, this data has often been used as an excellent example of regression to the mean.

Understanding Regression to the Mean

Whether overlooked or misexplained, the phenomenon of regression is foreign to the human mind. Regression was first recognized and understood two hundred years later than the theory of gravity and differential calculus. Moreover, it took one of the finest British minds of the 19th century to explain the regression.

This phenomenon was first described by Sir Francis Galton, Charles Darwin's second cousin, who had truly encyclopedic knowledge. In a paper entitled "Regression to the Mean in Inheritance," published in 1886, he reported measuring several successive generations of seeds and comparing the heights of children with the heights of their parents. He writes about seeds like this:

“The research yielded an interesting result, and on the basis of it, on February 9, 1877, I gave a lecture to the Royal Association. The experiments showed that the offspring did not resemble the parents in size, but always turned out to be more ordinary, that is, fewer large parents or more small ones... The experiments also showed that, on average, the regression of the offspring is directly proportional to the deviation of the parents from the mean.”

Galton apparently expected the learned audience at the Royal Association, the world's oldest independent research organization, to be as surprised by his "interesting results" as he was. But the most interesting thing is that he was surprised by the usual statistical pattern. Regression is ubiquitous, but we do not recognize it. She's hiding in plain sight. Within a few years, with the help of the eminent statisticians of his time, Galton went from the discovery of hereditary regression of size to the broader understanding that regression inevitably occurs when there is incomplete correlation between two quantities.

Among the obstacles the researcher had to overcome was the problem of measuring regression between quantities expressed in different units: for example, weight and ability to play the piano. They are measured by taking the entire population as a standard for comparison. Imagine that 100 children from all grades of primary school were measured for their weight and play ability and ranked the results in order, from the maximum to the minimum value of each indicator. If Jane is third in music and twenty-seventh in weight, you can say that she is better at playing the piano than she is tall. Let's make a few assumptions for simplicity.

Any age:

Success in playing the piano depends only on the number of hours of practice per week.

Weight depends solely on the amount of ice cream consumed.

Eating ice cream and the number of hours of music lessons per week are independent variables.

Now we can write some equations using list positions (or standard scores as statisticians call them):

weight = age + ice cream consumption playing piano = age + hours of practice per week

Obviously, when trying to predict piano performance by weight, or vice versa, regression to the mean will appear. If all we know about Tom is that he is a twelfth in weight (much above average), we can statistically conclude that Tom is probably older than average and probably consumes more ice cream than others. If all we know about Barbara is that she is eighty-fifth in piano (well below the group average), we can conclude that Barbara is most likely still young and probably practices less than others.

The correlation coefficient between two quantities, ranging from 0 to 1, is a measure of the relative weight of the factors influencing both of them. For example, we all share half of our genes with each of our parents, and for traits that have little external influence (such as height), the correlation between parent and child is close to 0.5. To evaluate the value of the correlation measure, I will give several examples of coefficients:

The correlation between the sizes of objects accurately measured in metric or imperial units is 1. All determining factors affect both measurements.

The correlation between self-reported weight and height for adult American men is 0.41. If women and children are included in the group, the correlation will be much higher because an individual's gender and age influence their assessment of their height and weight, which increases the relative values ​​of the common factors.

The correlation between high school academic ability tests and college GPA is approximately 0.60. However, the correlation between aptitude tests and graduate success is much lower - largely because the level of ability in this group does not vary much. If everyone's abilities are approximately the same, then the difference in this parameter is unlikely to greatly affect the measure of success.

The correlation between income and educational attainment in the United States is approximately 0.40.

The correlation between a family's income and the last four digits of their phone number is 0.

It took Francis Galton several years to understand that correlation and regression are not two different concepts, but two perspectives on one. The general rule is quite simple, but it has surprising consequences: in cases where the correlation is not perfect, regression to the mean occurs. To illustrate Galton's discovery, let's take a suggestion that many find quite curious:

Smart women often marry less intelligent men.

If at a party you ask your friends to find an explanation for this fact, then you are guaranteed an interesting conversation. Even people familiar with statistics will interpret this statement in causal terms. Some will think that smart women seek to avoid competition from smart men; someone will assume that they are forced to make compromises when choosing a spouse due to the fact that smart men do not want to compete with smart women; others will offer more far-fetched explanations. Now think about the following statement:

The correlation between spouses' intelligence scores is not perfect.

Of course, this statement is true - and completely uninteresting. In this case, no one expects perfect correlation. There is nothing to explain here. However, from an algebraic point of view, these two statements are equivalent. If the correlation between spouses' intelligence scores is not perfect (and if women and men do not differ on average in intelligence), then it is mathematically inevitable that intelligent women will marry men who are, on average, less intelligent (and vice versa). Observed regression to the mean cannot be more interesting or more explainable than non-ideal correlation.

One can sympathize with Galton - attempts to understand and explain the phenomenon of regression are not easy. As statistician David Friedman ironically notes, if the issue of regression arises in a trial, the party that has to explain it to the jury is sure to lose. Why is this so difficult? The main reason for the difficulty is mentioned regularly in this book: our minds are prone to causal explanations and do not cope well with “simple statistics.” If some event attracts our attention, associative memory begins to look for its cause, or rather, any reason already stored in memory is activated. When regression is discovered, causal explanations are sought, but they will be incorrect, because in fact regression to the mean has an explanation, but there are no reasons. One of the things that comes to our attention during golf tournaments is that athletes who play well on the first day often play worse afterwards. The best explanation is that these golfers got unusually lucky on the first day, but that explanation lacks the force of causality that our minds prefer. We pay good money to those who come up with interesting explanations for regression effects for us. A commentator on a business news channel who correctly remarks that “this year was better for business because last year was bad” will likely not last long on the air.

Our difficulties in understanding regression arise from both System 1 and System 2. Without further instruction (and in many cases, even after some familiarity with statistics), the relationship between correlation and regression remains unclear. It is difficult for System 2 to understand and internalize it. This is partly due to System 1's insistence on providing causal explanations.

Three months of using energy drinks to treat depression in children produces significant improvements.

I made up this headline, but what it describes is true: Giving energy drinks to depressed children over a period of time shows clinically significant improvement. Similarly, children with depression who stand on their heads for five minutes or pet cats for twenty minutes every day will also show improvement. Most readers of such headlines will automatically conclude that the improvement was due to the energy drink or petting the cat, but this is a completely unfounded conclusion. Depressed children are an extreme group, and such groups regress toward the mean over time. The correlation between depression levels across successive tests is imperfect, so regression to the mean is inevitable: Children with depression will get a little better over time, even if they don't pet cats or drink Red Bull. To conclude that an energy drink - or any other treatment - is effective, it is necessary to compare a group of patients receiving it with a control group receiving no treatment at all (or, better yet, a placebo). The control group is expected to show improvement due to regression alone, and the purpose of the experiment is to find out whether patients receiving treatment improve more than is explained by regression.

Incorrect causal attributions of the regression effect are not limited to readers of the popular press. Statistician Howard Weiner compiled a long list of prominent researchers who made the same mistake, that is, confusing correlation with causation. The regression effect is a common source of problems in research, and experienced scientists develop a healthy fear of pitfalls, that is, unwarranted causal inferences.

One of my favorite examples of error in intuitive predictions comes from Max Bazerman's excellent book, Value Judgments in Management Decision Making, and is adapted from:

You are forecasting sales in a chain of stores. All stores in the chain are similar in size and assortment, but their sales volume varies due to location, competition and various random factors. You were presented with the results for 2011 and asked to determine sales in 2012. You are instructed to stick to economists' general forecast that overall sales growth will be 10%. How would you complete the following table?

After reading this chapter, you know that the obvious solution to add 10% to each store's sales is wrong. The forecast should be regressive, that is, for stores with poor results you should add more than 10%, and for the rest - less, or even subtract something. However, most people are puzzled by this task: why ask about the obvious? As Galton discovered, the concept of regression is not obvious.