Scientists' parochialism about their method for gaining knowledge (the scientific method) can provoke resentment. I recall some rousing oratory in an undergrad anthropology course about "non-Western science". Even closer to home, I once seriously offended a relative who works in the public school system by suggesting undiplomatically that her thoughts about dyslexia were interesting but that couldn't be taken too seriously until they had been subjected to empirical study.
My tactlessness aside, there are very good reasons to be skeptical of empirical "knowledge" that has not been rigorously tested in a controlled manner. There are many such "alternative routes to wisdom": Scripture, psychics, mediums... -- but the one that I want to specifically debunk in this post is experience. Experience was at the heart of my disagreement with the veteran teacher, who felt her knowledge extracted from the years was more than enough evidence for her ideas. Experience, I believe, is related to why many people believe psychics ("her predictions are always right!") or even Scripture. And experience, it turns out, isn't all it's cracked up to be.
A recent study found that experts weren't necessarily much better than laypeople in predicting the outcomes of disputes. The disputes "included a hostile takeover attempt, nations preparing for war, a controversial investment proposal, a nurses’ strike, an action by football players for a larger share of the gate, an employee resisting the downgrading of her job, artists demanding taxpayer funding, and a new distribution arrangement that a manufacturer proposed to retailers." Experts were asked to make predictions about a particular conflict chosen to match their specialty. Not only were the experts barely better than non-experts, they were barely better than someone guessing randomly.
This isn't actually news, as scientists have suspected this for some time. Mutual funds managers are high-paid investing experts, but it is rare when a fund manager manages to beat the market.
Why doesn't experience help as much as it should? And why do we trust experience? In fact, the failure of expertise isn't all that surprising based on what we know about human memory and learning. People tend to ignore evidence that doesn't fit their beliefs about the world but remember positive evidence strongly. For instance, many people are convinced that the best way to make sure it won't rain is to take an umbrella with them on the way to work. Even without a controlled experiment, I'm fairly sure Sally Brown's umbrella habits don't control the weather. What is probably going on is that she remembers vividly all those times that she lugged around a useless umbrella, but she's completely forgotten all the times it did rain.
Studies of sports gamblers picking winning teams (Gilovich, 1983; Gilovich & Douglas, 1986) found that they tended to remember incorrect picks not as poor judgment on their part, but near-wins. "I was correct in picking St. Louis. They would have won except for that goofy bounce the ball took after the kickoff..." Evidence that should have proved their theories wrong were explained away instead.
Thus, accumulated wisdom is likely to be misleading. You may remember that Madame Juzminda's predictions always come true, but if you want to be sure, you shouldn't trust your memory but written record. You say you can predict which baseball rookies will have meteoric careers and which will fizzle? That's an empirical claim, and we can easily test it scientifically. Billy Beane became a baseball sensation by jettisoning "experience" and relying on fact. I, for instance, believe that I'm better at predicting the rain than Accuweather, but until I've actually proven it in a controlled study, you're probably better off with the real weather report (although, just for kicks, I am planning on testing my weather acumen against the experts and will blog on the results in a month or two).
This is not all to say that Madame Juzminda, the experienced doctor or Scripture are all wrong. Maybe they are. Maybe they aren't. Until you've tested them, it's better to keep an open mind.
(Caveat: Scientists have egos, bad memories and agendas. The scientific method is an ideal not always reached. So's democracy, but it still beats a dictatorship any day.)
My tactlessness aside, there are very good reasons to be skeptical of empirical "knowledge" that has not been rigorously tested in a controlled manner. There are many such "alternative routes to wisdom": Scripture, psychics, mediums... -- but the one that I want to specifically debunk in this post is experience. Experience was at the heart of my disagreement with the veteran teacher, who felt her knowledge extracted from the years was more than enough evidence for her ideas. Experience, I believe, is related to why many people believe psychics ("her predictions are always right!") or even Scripture. And experience, it turns out, isn't all it's cracked up to be.
A recent study found that experts weren't necessarily much better than laypeople in predicting the outcomes of disputes. The disputes "included a hostile takeover attempt, nations preparing for war, a controversial investment proposal, a nurses’ strike, an action by football players for a larger share of the gate, an employee resisting the downgrading of her job, artists demanding taxpayer funding, and a new distribution arrangement that a manufacturer proposed to retailers." Experts were asked to make predictions about a particular conflict chosen to match their specialty. Not only were the experts barely better than non-experts, they were barely better than someone guessing randomly.
This isn't actually news, as scientists have suspected this for some time. Mutual funds managers are high-paid investing experts, but it is rare when a fund manager manages to beat the market.
Why doesn't experience help as much as it should? And why do we trust experience? In fact, the failure of expertise isn't all that surprising based on what we know about human memory and learning. People tend to ignore evidence that doesn't fit their beliefs about the world but remember positive evidence strongly. For instance, many people are convinced that the best way to make sure it won't rain is to take an umbrella with them on the way to work. Even without a controlled experiment, I'm fairly sure Sally Brown's umbrella habits don't control the weather. What is probably going on is that she remembers vividly all those times that she lugged around a useless umbrella, but she's completely forgotten all the times it did rain.
Studies of sports gamblers picking winning teams (Gilovich, 1983; Gilovich & Douglas, 1986) found that they tended to remember incorrect picks not as poor judgment on their part, but near-wins. "I was correct in picking St. Louis. They would have won except for that goofy bounce the ball took after the kickoff..." Evidence that should have proved their theories wrong were explained away instead.
Thus, accumulated wisdom is likely to be misleading. You may remember that Madame Juzminda's predictions always come true, but if you want to be sure, you shouldn't trust your memory but written record. You say you can predict which baseball rookies will have meteoric careers and which will fizzle? That's an empirical claim, and we can easily test it scientifically. Billy Beane became a baseball sensation by jettisoning "experience" and relying on fact. I, for instance, believe that I'm better at predicting the rain than Accuweather, but until I've actually proven it in a controlled study, you're probably better off with the real weather report (although, just for kicks, I am planning on testing my weather acumen against the experts and will blog on the results in a month or two).
This is not all to say that Madame Juzminda, the experienced doctor or Scripture are all wrong. Maybe they are. Maybe they aren't. Until you've tested them, it's better to keep an open mind.
(Caveat: Scientists have egos, bad memories and agendas. The scientific method is an ideal not always reached. So's democracy, but it still beats a dictatorship any day.)
No comments:
Post a Comment