On our American episode this week, while talking about Colombian oil exploration outfit GeoPark, we had reason to mention the classic value investing book “What Works on Wall Street: A Guide to the Best-Performing Investment Strategies of All Time” by James P. O’Shaughnessy (his cousin was a co-founder of GeoPark — their grandfather was an OG American wildcatter).
This prompted me to pull out WWOWS again and review it. This caught my eye.
In Chapter 2 “The Unreliable Experts: Getting in the Way of Outstanding Performance”, he talks about decision making, and the two basic models people use to make decisions.
“Generally, there are two ways to make predictions. Most common is for a person to run through a variety of possible outcomes in his or her head, essentially relying on knowledge, experience, and common sense to reach a decision. This is known as a “clinical” or intuitive approach, and is the way traditional active money managers make choices. The stock analyst may pore over a company’s financial statements; interview management; talk to customers and competitors; and finally try to make an overall forecast. The graduate school administrator might use a host of data, from college grade point average to interviews with applicants, to determine if students should be accepted. This type of judgment relies on the perceptiveness of the forecaster.
The other way to reach a decision is the actuarial, or quantitative, approach. Here, the forecaster makes no subjective judgments. Empirical relationships between the data and the desired outcome are used to reach conclusions. This method relies solely on proven relationships using large samples of data. The graduate school administrator might use a model that finds college grade point average highly correlated to graduate school success and admit only those who have made a certain grade. In almost every instance, from stock analysts to doctors, we naturally prefer qualitative, intuitive methods. In most instances, we’re wrong.
Jack Sawyer, a researcher who published a review of 45 studies comparing the two forecasting techniques: In none was the clinical, intuitive method—the one favored by most people-found to be superior. What’s more, Sawyer included instances where the human judges had more information than the model and were given the results of the quantitative models before being asked for a prediction. The human judges still failed to beat the actuarial models!”
As the Sawyer study was done in 1966, and is a little long in the tooth, I did some digging into this theory that models out-perform human intuition, and found that while the original Sawyer review covered 45 studies, modern meta-analyses now encompass hundreds of cases across medicine, psychology, finance, and criminal justice, consistently showing that mechanical/actuarial models outperform human intuition by a margin of 10% to 15% on average.
However, there are caveats.
- The “Broken Leg” Rule , developed by Paul Meehl, an American clinical psychologist. He pointed out that the most significant exception is when a human possesses a single, high-validity piece of information that the model was never designed to account for. Example: If a model predicts a person will go to the cinema tonight based on a five-year pattern, but you know that person just broke their leg, you should overrule the model. The catch, however, is that “Experts” tend to see “broken legs” everywhere. They mistake “interesting” or “salient” information for “predictive” information. To beat the model, the human must only intervene for rare, high-impact facts that truly negate the base rate.
- High-Validity Environments (Kahneman & Klein). Daniel Kahneman (who championed the model-is-better view) and Gary Klein (who championed expert intuition) found a truce. They concluded that human intuition can be superior, but only if two conditions are met: The environment must have stable, predictable regularities (e.g., firefighting, chess, anesthesiology). And the expert must have had years of practice with immediate, high-quality feedback. Unfortunately, they concluded that in “noisy” environments like psychiatry, political forecasting, or stock picking, there is no “regularity” for the brain to learn, and models remain undefeated.
The bottom line for us as investors is that there’s a very high probability that our intuition is almost always going to be wrong.
What works best over the long term is a model that tells us what to do.
I might think I know better than the model. My gut feeling might tell me I shouldn’t invest in, say, APOLLO TOURISM because they have let me down time after time (a reference our long-time listeners will understand).
I should ignore my gut feeling.
That’s probably the hardest lesson to learn. Our brains suck at making decisions. Listen, brains are great. Nice work, evolution. Sure, it took millions of years and lots of side quests, but you did something pretty cool.
But a model will beat a brain almost every time.
Yeah yeah, the brains built the models. Sure. So the brains win in the end.
But only if they let go of their feeling of superiority and give in to the models.
QAV Myth Killers is a weekly column in the QAV newsletter, taking apart a piece of
investing conventional wisdom. Read the series, or
get it by email every Friday.
General advice only. Spacecraft Publishing Pty Ltd
trading as QAV is a Corporate Authorised Representative (CAR 001292718) of MF & Co.
Asset Management Pty Ltd (AFSL 520442). This is general information and does not take your
personal circumstances into account. See our
disclosure.