I’ve recently been reading a fascinating book called “Psychology of Intelligence Analysis” (1999), written by Richards Heuer, who worked for the CIA for 45 years. He wrote this book initially for internal use at the CIA to provide a methodology for overcoming intelligence biases by using a framework for decision-making. Heuer came up with an analytic process known as Analysis of Competing Hypotheses (ACH) to improve the intelligence work done inside the agency. Sadly, it wasn’t enough to help them prevent the 9/11 attacks or to avoid the quagmires the U.S.A. found themselves in during the wars in Iraq and Afghanistan — but that’s beside the point.
One of the interesting points the book makes is that human minds are poorly wired to cope effectively with inherent uncertainty.
Chapter 5 really jumped out at me as having implications for investing. In that chapter, he talks about experiments done by psychologists to test whether experts provided with more information make superior decisions.
What they discovered was, once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her
estimates. Additional information does, on the other hand, lead the analyst to become more confident in their judgment, to the point of over-confidence.
As it turns out, experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. In other words, analysts actually use much less of the available information than they think they do.
In one experiment that TK will love, eight experienced horserace handicappers were shown a list of 88 variables found on a typical past-performance chart, eg the weight to be carried, the percentage of races where the horse finished first, second or third, the jockey’s record, etc. Each handicapper was asked to identify what he considered to be the five most important items of information, those that he would use to handicap a race if he were limited to only five bits of data per horse. Each was then asked to select the 10, 20 and 40 most important variables they would use. Then they were given true data that had been sterilised so the horses in actual races couldn’t be identified for 40 past races and were then asked to rank the top five horses in each race in order of expected finish. Each handicapper was given the data in increments of the 5, 10, 20 and 40 variables that they had judged to be most useful. And each one predicted each race four times, once with each of the four different levels of information. For each prediction, each handicap were assigned a value from 0 to 100% to indicate their degree of confidence in the accuracy of their prediction. When their predictions were compared with the actual outcomes of these 40 races, it turned out that the average accuracy of predictions remained the same, regardless of how much information they had available.
Three of the handicappers actually showed less accuracy as the amount of information increased. Two improved their accuracy and three were unchanged. What’s fascinating though is that all of them expressed increased confidence in their judgments as they were given more information to work with. When they were only working with five items of information their confidence was pretty well calibrated to their accuracy but the more information they were given the more overconfident they became.
Other experiments have shown the same relationships between the amount of information, accuracy and analyst confidence in other fields. There was one experiment with clinical psychologists where 32 psychologists with varying levels of experience were made to ask, made to make judgments about the life of a relatively normal individual. As in the handicapping experiment, the more information they had, the more confident they were about their analysis even though there was a negligible increase in accuracy. Another series of experiments worked with medical doctors diagnosing illnesses. Same results.
So what, you might be asking, does any of this have to do with QAV?
Over the years we’ve been producing the show it’s always struck me how well the model performs by using such a relatively limited set of data. We don’t spend hours, days or weeks going deep on a particular company’s financials, or management, or their sales pipeline, their R&D, or their competitive landscape, their culture or their accounting practices. Lots of fund and firms, on the other hand, employ lots of analysts who do those sorts of things. Guess what? We outperform most of them. There might be a range of reasons we are able to do that, but I suspect one of them is that Tony designed a system which looks at a limited range of data points and, from those, is able to make a pretty accurate prediction whether or not the stock will outperform the rest of the market. The system doesn’t always get it right, of course, but then we have other rules that accommodate for those scenarios.
I used to have a sticker on the water bottle I take to kung fu which said “Hold On — Let Me Overthink This”, because my Sifu was always telling me “Cameron — don’t overthink it” (you have to imagine that in a Scottish accent). Kung fu, like QAV, is built around a series of techniques which, if applied with discipline, will deliver a good result most of the time. My brain likes to go down rabbit holes — “what about if this happens” and “what if they do X instead of Y”? I’ve had to learn not to do that. I’ve had to learn to trust the system. It works. Generations of kung fu masters before me have designed a system that works — most of the time.
We are only in the second generation of QAV (if TK is the first and we are all the second), but he learned from the black belts that came before him. Sifu Buffett and Sifu Munger. They learned it from Sifu Graham.
I know, I know. More kung fu analogies. I can’t help myself.
Don’t overthink it.
QAV Myth Killers is a weekly column in the QAV newsletter, taking apart a piece of
investing conventional wisdom. Read the series, or
get it by email every Friday.
General advice only. Spacecraft Publishing Pty Ltd
trading as QAV is a Corporate Authorised Representative (CAR 001292718) of MF & Co.
Asset Management Pty Ltd (AFSL 520442). This is general information and does not take your
personal circumstances into account. See our
disclosure.