Here is another insightful experiment from Tversky and Kahneman:
It's not clear from the description whether the instructors considered whether their lesson plan would be beneficial to morale, but in any case, they were almost certainly wrong. They fell for a statistical phenomenon known as "regression to the mean."
Basically, every time you measure something, there is some error. For instance, although Sally may be a B student, sometimes she gets As on her tests and sometimes she gets Cs. Some days she has good days and some days she has bad days.
Now suppose you give a test to the whole class and then select all the students who got As to be in a special program. That group of students who got As will include some who are normally A students, but it will also include some people who normally are B or C students but who had a good day. So, if you were to re-test those same students, the average grade would decline, perhaps to an A- or B+. This isn't because the students got stupider; it's just that the students who got fluke As the first time are unlikely to repeat their performance.
Similarly, if you had picked all the students who failed the exam, that group of students would have included both true F students as well as a few C or D students (maybe even B or A students) who were having a rough day. If you retest them, the average grade will move up, because those C and D students will likely do better the second time. They haven't gotten smarter; it's just regression to the mean.
Those flight students would probably have had a better experience if their instructors knew about regression to the mean.
----
Some readers might have wondered the following: if, in the group of students who got As, some of their scores will go down upon re-testing, shouldn't some go up? Yes and no. Some scores would go up, but those are mostly typically A students who got Bs or Cs on that particular exam. However, you have already excluded them from the group, so their rebounding scores can't off-set the falling scores of the typically B and C students.
Tversky, A., Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science, 185(4157), 1124-1131.
In a discussion of flight training, experienced instructors noted that praise for an exceptionally smooth landing is typically followed by a poorer landing on the next try, while harsh criticism after a rough landing is usually followed by an improvement on the next try. The instructors concluded that verbal rewards are detrimental to learning, while verbal punishments are beneficial.
It's not clear from the description whether the instructors considered whether their lesson plan would be beneficial to morale, but in any case, they were almost certainly wrong. They fell for a statistical phenomenon known as "regression to the mean."
Basically, every time you measure something, there is some error. For instance, although Sally may be a B student, sometimes she gets As on her tests and sometimes she gets Cs. Some days she has good days and some days she has bad days.
Now suppose you give a test to the whole class and then select all the students who got As to be in a special program. That group of students who got As will include some who are normally A students, but it will also include some people who normally are B or C students but who had a good day. So, if you were to re-test those same students, the average grade would decline, perhaps to an A- or B+. This isn't because the students got stupider; it's just that the students who got fluke As the first time are unlikely to repeat their performance.
Similarly, if you had picked all the students who failed the exam, that group of students would have included both true F students as well as a few C or D students (maybe even B or A students) who were having a rough day. If you retest them, the average grade will move up, because those C and D students will likely do better the second time. They haven't gotten smarter; it's just regression to the mean.
Those flight students would probably have had a better experience if their instructors knew about regression to the mean.
----
Some readers might have wondered the following: if, in the group of students who got As, some of their scores will go down upon re-testing, shouldn't some go up? Yes and no. Some scores would go up, but those are mostly typically A students who got Bs or Cs on that particular exam. However, you have already excluded them from the group, so their rebounding scores can't off-set the falling scores of the typically B and C students.
Tversky, A., Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science, 185(4157), 1124-1131.
2 comments:
Ok. You are saying that the experience of the pilot instructors is similar to a teacher who praises students who got an A-B on their most recent test and yells at students who got an D-F and then notes that the F students are more likely to have raised their grades on the next test than that the B students have raise theirs. Consequently the teacher decides that negative reinforcement is better than positive. Is this correct?
So, just how would you decide if negative reinforcement were better or worse than positive reinforcement other than accepting the "we all know" position?
The problem with the pilot instructors' "experiment" was that they chose a biased sample. A better test would be to try either negative or positive reinforcement on a random sample.
Even then, it is always tricky to generalize from one experiment, since it's always possible your "random" sample was biased in some way, or that your particular form of positive or negative reinforcement was the key. So it would be important to do a series of experiments.
Post a Comment