Decision Skills
Are you a bad decision-maker?
The golden rule of decision analysis: avoid outcome bias
Experts in decision analysis have a nifty trick for sniffing out unskilled decision-makers. Curious to know what they’d say about you? Find out by revisiting our quiz from last week with a follow-up question.
Quiz time!
Heather is Canadian, kind, friendly, smart, and loves animals. She is a sustainability consultant. In college, she studied math and psychology. She likes going for long walks and lives near several hiking trails.
Which is more probable about Heather?
- Heather has a PhD.
- Heather has a PhD and owns a dog.
Make sure you’ve prepared your answer before you scroll down!
Want to read this story later? Save it in Journal.
If you chose option 1 (Has a PhD), congratulations! You avoided the conjunction fallacy. Option 1 is the better answer because option 2 gives you more opportunities to be wrong. To math lovers, it’s as obvious as P(A) ≥ P(A ∩ B). For an in-depth explanation of the conjunction fallacy and why your stereotypes can lead you to make excessively elaborate conclusions, check out my article, Don’t fall for the conjunction fallacy! (Go on, take a look, I’ll wait until you’re done.)
That article said all kinds of things about biases, stereotypes, science, and how not to be a bad analyst. What it didn’t say was whether Heather actually owns a dog.
So, does she?
Now that you know Heather owns a dog, which of these options best describes your feelings about your earlier choice?
Reminder: option 1 was (Has a PhD) and option 2 was (Has a PhD and owns a dog).
A) “I originally chose option 1, but I should have chosen option 2 instead.”
B) “I originally chose option 2, but I should have chosen option 1 instead.”
C) “I originally chose option 1. It was the wiser choice. I was right to pick it.”
D) “I originally chose option 2. It was the wiser choice. I was right to pick it.”
Did you choose C?
Experts in decision science would rank your answers from best to worst like this: C, B, A, D. Yes, every choice that isn’t C spells BAD here. Why?
Good decision-makers don’t fall for outcome bias!
The first principle of decision analysis is: avoid outcome bias. It’s what we teach students on day 1. Immunity to outcome bias is the minimum requirement for basic decision competence. There’s plenty to learn on the journey to good decision-making, but it’s hopeless to proceed until you’ve cleared this first hurdle.
(If you’re feeling impatient, scroll down to find out what your choice says about you.)
What is outcome bias?
In psychology, outcome bias is the error of evaluating a decision based on things that you couldn’t have known at the time. Don’t confuse outcome bias with hindsight bias (the one where you adjust your memory after learning the answer, saying “I knew it all along” although you didn’t).
If you want a superb example of outcome-biased thinking, consider evaluating the Swedish government’s COVID-19 decisions. The difference between the general public and decision professionals is that the (outcome-biased) public will judge Sweden based on “how things turn out” while decision professionals will not. Though decision scientists may differ in our appraisals of the situation, we will only judge Sweden based on what was known at the time the decisions were made.
Many of you will be surprised to hear this, since outcome bias is a socially-acceptable form of mass irrationality. You might even feel the beginnings of rage, especially if you’ve been brought up to believe that “how things turn out” is exactly the way to evaluate decision quality. Too see why this is an illusion, turn to my article “The problem with analyzing policy decisions in hindsight.”
The core principle of decision analysis
The objective of this article is to teach you the most fundamental principle in decision analysis: The quality of a decision should be evaluated using only the information available to the decision-maker at the time the decision was made.
The quality of a decision should be evaluated using only the information available to the decision-maker at the time the decision was made.
In other words, don’t let hindsight mess with you. (Especially in 2020.)
A decision expert always separates decisions from outcomes.
But let’s get back to Heather and her dog.
What your choice says about you
Choice C
Good choice! You answered the first question wisely and recognized that it was the best choice based on what you knew at the time, so you felt confident about your decision process.
You’ve got great instincts!
When shown the puppy photo, you didn’t fall for outcome bias. You’d make a similar decision the same way, just like a trained decision scientist would. Congratulations!
(This doesn’t necessarily mean you’re a good decision-maker, since that’s a long, complicated journey that takes training and talent. But you’ve sailed over the first hurdle.)
Choice B
Although you answered incorrectly the first time, I’m thrilled for you: choosing B tells me that you’re someone who learns from your mistakes… in the right way.
You learn from your mistakes… correctly!
You didn’t switch because of the puppy photo outcome, but because you understood why you shouldn’t fall for the conjunction fallacy and updated the error in your decision process. Things aren’t all that B-A-D with you. To be honest, you warm my heart more than the Choice C folks because I just saw you grow right before my eyes. You learned and I love you for it!
Choice A
The good news is that your instincts led you to the better decision the first time around. That’s all the good news you get; the rest is bad news. Even though you started off on the right foot, now you’re regretting it. Wishing you had made a worse decision with a better outcome is unwise. You’re showing susceptibility to outcome bias and letting hindsight teach you the wrong lessons about your decision process.
You threaten society’s ability to promote and retain competent leaders.
Learning from your mistakes is a good thing, but don’t learn the wrong lessons by misdiagnosing your mistakes. Sometimes, life throws you a random curveball you couldn’t have anticipated no matter your level of decision genius. When that happens, adjusting your decision process is bad thinking. It’s the sort of bad thinking that threaten’s society’s ability to promote and retain competent leaders.
To understand why, follow this link for an example.
Choice D
You insist on aggressively missing the point, displaying a double dose of bias. Like the person who went for Choice B, you fell for the conjunction fallacy, letting your stereotypes push you into making the more elaborate conclusion without evidence… while thinking your conclusion was less elaborate because it fit your stereotypes. Unlike folks who picked Choice B, instead of learning from a critique of your decision process, you used the outcome to exclaim, “See? I told you so! She has a dog. I was right to do what I did and I’ll do it the same way next time.”
A double dose of bias!
No, you were lucky.
You made the decision badly and just so happened to have a good outcome. Now you’re failing to understand the difference between luck and skill. You’re displaying the catlike reflexes of a clumsy pedestrian who narrowly avoids collision with a bicycle by randomly tripping over their own feet (without ever seeing the bicycle coming). “I meant to do that!” Yeah, right, you superspy you.
It’s like not understanding that there’s difference between luck and skill.
Like Choice A, this kind of reasoning shrinks society’s supply of skilled decision-makers, but it goes a step further: mixed with a few other biases, it perpetuates prejudice.
This style of decision-making is unaware of its biases and aggressively perpetuates prejudice.
That discussion is worth its own set of books, so I’ll only give you a short summary along with the keywords you can look up if you’re interested.
Poor decision skills perpetuate prejudice
If you don’t train yourself into better decision-making habits, psychologists predict that you’re more likely to:
- Overestimate the likelihood of exemplars you perceive as stereotypical. (Availability bias.)
- Ignore information that challenges your preexisting beliefs, including information that conflicts with your stereotypes. (Confirmation bias.)
- Come to far-fetched conclusions based on stereotypes that “feel right” despite being unwarranted by your evidence. (Conjunction fallacy.)
- Ignore your decision process and overfocus on outcomes. (Outcome bias.)
- Take credit if you’re lucky enough to get a good outcome, but blame someone/something else if you get a bad outcome. (Self-serving bias.)
Do you see the feedback loop?
What to do instead
- Availability bias is a symptom of imperfect human memory, so rely on something with better-than-human memory: a machine (or a pencil). Where possible, favor statistically-unbiased written records over human memory.
- Where there’s uncertainty, keep an open mind and take measures to reduce confirmation bias.
- Practice extreme reluctance to jump to conclusions, especially when they’re more elaborate than your evidence justifies. (Even if the elaborate conclusion “feels” right.)
- Evaluate decisions and decision-makers based only on what was known at the time of their decision, ignoring the outcome.
- Evaluate risk before the decision is taken and use risk magnitude to apportion credit/blame between the decision-maker and factors outside their control. Don’t blame decision-makers for the outcomes of their gambles, but do blame them for taking inappropriate gambles in the first place… even if they luck out. Assign blame whenever a decision-maker displays poor decision skills, including: (i) putting in less effort than the stakes warrant; (ii) taking on an excessive amount of risk; (iii) ignoring available information; (iv) incorporating available information incorrectly.
If you’re keen to learn more, check out my follow-up article, The problem with analyzing policy decisions in hindsight.
Thanks for reading! Liked the author?
If you’re keen to read more of my writing, most of the links in this article take you to my other musings. Can’t choose? Try this one:
Connect with Cassie Kozyrkov
Let’s be friends! You can find me on Twitter, YouTube, and LinkedIn. Interested in having me speak at your event? Use this form to get in touch.
Thanks for reading! How about a course?
If you had fun here and you’re looking for an applied AI course designed to be fun for beginners and experts alike, here’s one I made for your amusement:
