*Updated with More Analysis*
So says Philip Babcock in today's New York Times. He claims:
So it's important to show that individual schools have lowered their standards, and that this is true for the selective schools as well as the not-selective schools. The article links to this study by Babcock. This study analyzes a series of surveys of student study habits from the 1960s to the 2000s, and thus seems to be the basis of his argument, and in fact the introduction contains almost the identical statement that I have quoted above. Nonetheless, despite these strong conclusions, the data that would support them appear to be missing.
What Babcock might have meant, and more problems with the data
It is possible that when Babcock was saying that the decrease in study time was true of all types of schools, he meant that when you look at all types of schools in 2003/4, students at all levels report studying less than the average student reported in 1961. The problem is that, for all we know, the schools in his sample were more selective in 1961 than they were in 2003/4.
Moreover, the is something worrisome about his selectivity data. Whenever analyzing data, many researchers like to do what is called a "sanity check": they make sure that the data contain results that are known to be true. If you were looking at a study of different types of athletes, you might make sure that the jockeys are shorter than the basketball players, lighter than the football players and chew less tobacco than the baseball players. If you find any of these things do not hold, you might go back and make sure there isn't a type-o somewhere in your data-entry process.
I worry that Babcock's data fail the sanity check. Specifically, look at the number of hours studies according to selectivity of school in 2003:
highly selective: 13.47 hours
middle: 14.68 hours
non-selective: 16.49 hours
Note that this effect is larger than the decline in number of hours studied between 1988 and 2003, so in terms of this dataset, this is a large effect (again, I cannot tell if it is significant, because the relevant statistical information is not provided) and it's not in the direction one would think. I will admit that it is possible that students at highly selective schools really do study less than the folks at JuCo, but that conflicts heavily with my pretty extensive anecdotal database. So either a) the world is very different from how I thought it was -- in which case, I want more evidence than just this survey -- b) Babcock has defined selectivity incorrectly, or c) there is something wrong with these data.
One last worrisome fact
I considered the possibility that the data Babcock was quoting were in a different paper. The only other paper on Babcock's website that looked promising was this American Enterprise Institute report. This is not a research paper, but rather summarizes research. Specifically, according to footnote #2, it summarizes the research in the paper I just discussed. Strangely, this paper does have a graph (Figure 4) breaking down study habits of students in the 1960s based on selectivity of the school they are attending: the very data he states do not exist in the later paper.
I'm not really sure what to make of that, and have nothing further to say on the topic. At the very least, I would be hesitant to use those graphs as evidence to support the general claim that study habits have changed even at the selective schools, since it's unclear where the data case from, or if in fact they even exist (to be clear: it's Babcock who says they don't exist, not me).
Conclusion
To summarize, there seems to be very little evidence to support Babcock's conclusion that study time has decreased even at selective schools by 10 hours from the 1960s to modern day. That is, he has a survey from 1961 in which students studied 25 hrs/week, two surveys in the 1980s in which students studied 17 hours/week, and two surveys in the 2000s in which students studied 14-15 hrs/week, but these surveys are all based on different types of students at different schools, so it's hard to make any strong conclusions. If I compared the weight of football places from Oberlin in 1930 and Ohio State in 2005, I'd find a great increase in weight, but in fact the weight of football players at Oberlin probably has not increased much over that time period.
Moreover, there are aspects of these data that deserve some skepticism. When report to people who went to selective schools that these data suggest students at such schools study 13 hrs/week, the response is usually something like, "Do you mean per day?"
Finally, since no statistics were run, it's quite possible that none of the results in this study are significant.
I want to be clear that I'm not saying that Babcock's claims aren't true. I'm just saying that it's not clear he has any evidence to support them (which is not to say I think it's a bad study: it was a good study to have done and clearly took a lot of work, but I find it at best suggestive of future avenues of research and certainly not conclusive).
So says Philip Babcock in today's New York Times. He claims:
Full-time college students in the 1960s studies 24 hours per week, on average, and their counterparts today study 14 hours per week. The 10-hour decline is visible for students from all demographic groups and of all cognitive abilities, in every major and at every type of college.The claim that this is true for "every type of college" is important because he wants to conclude that schools have lowered their standards. The alternative is that there are more, low-quality schools now, or that some schools have massively lowered their standards. These are both potentially problems -- and are probably real -- but are not quite the same problem as all schools everywhere lowering their standards.
So it's important to show that individual schools have lowered their standards, and that this is true for the selective schools as well as the not-selective schools. The article links to this study by Babcock. This study analyzes a series of surveys of student study habits from the 1960s to the 2000s, and thus seems to be the basis of his argument, and in fact the introduction contains almost the identical statement that I have quoted above. Nonetheless, despite these strong conclusions, the data that would support them appear to be missing.
SAT scores and size are not available in the early years, so study time by college selectivity is not reported.He goes on to say that he can look at selectivity in the more recent surveys: specifically matched 1988-2003 surveys. These do show a decrease in study time from on the order of 1-2 hours for high-, medium- and low-selectivity schools (I cannot find how selectivity was defined). Whether this is even statistically significant is unclear, as he does not report any statistics or confidence intervals. In any case, it is not a 10 hour difference.
What Babcock might have meant, and more problems with the data
It is possible that when Babcock was saying that the decrease in study time was true of all types of schools, he meant that when you look at all types of schools in 2003/4, students at all levels report studying less than the average student reported in 1961. The problem is that, for all we know, the schools in his sample were more selective in 1961 than they were in 2003/4.
Moreover, the is something worrisome about his selectivity data. Whenever analyzing data, many researchers like to do what is called a "sanity check": they make sure that the data contain results that are known to be true. If you were looking at a study of different types of athletes, you might make sure that the jockeys are shorter than the basketball players, lighter than the football players and chew less tobacco than the baseball players. If you find any of these things do not hold, you might go back and make sure there isn't a type-o somewhere in your data-entry process.
I worry that Babcock's data fail the sanity check. Specifically, look at the number of hours studies according to selectivity of school in 2003:
highly selective: 13.47 hours
middle: 14.68 hours
non-selective: 16.49 hours
Note that this effect is larger than the decline in number of hours studied between 1988 and 2003, so in terms of this dataset, this is a large effect (again, I cannot tell if it is significant, because the relevant statistical information is not provided) and it's not in the direction one would think. I will admit that it is possible that students at highly selective schools really do study less than the folks at JuCo, but that conflicts heavily with my pretty extensive anecdotal database. So either a) the world is very different from how I thought it was -- in which case, I want more evidence than just this survey -- b) Babcock has defined selectivity incorrectly, or c) there is something wrong with these data.
One last worrisome fact
I considered the possibility that the data Babcock was quoting were in a different paper. The only other paper on Babcock's website that looked promising was this American Enterprise Institute report. This is not a research paper, but rather summarizes research. Specifically, according to footnote #2, it summarizes the research in the paper I just discussed. Strangely, this paper does have a graph (Figure 4) breaking down study habits of students in the 1960s based on selectivity of the school they are attending: the very data he states do not exist in the later paper.
I'm not really sure what to make of that, and have nothing further to say on the topic. At the very least, I would be hesitant to use those graphs as evidence to support the general claim that study habits have changed even at the selective schools, since it's unclear where the data case from, or if in fact they even exist (to be clear: it's Babcock who says they don't exist, not me).
Conclusion
To summarize, there seems to be very little evidence to support Babcock's conclusion that study time has decreased even at selective schools by 10 hours from the 1960s to modern day. That is, he has a survey from 1961 in which students studied 25 hrs/week, two surveys in the 1980s in which students studied 17 hours/week, and two surveys in the 2000s in which students studied 14-15 hrs/week, but these surveys are all based on different types of students at different schools, so it's hard to make any strong conclusions. If I compared the weight of football places from Oberlin in 1930 and Ohio State in 2005, I'd find a great increase in weight, but in fact the weight of football players at Oberlin probably has not increased much over that time period.
Moreover, there are aspects of these data that deserve some skepticism. When report to people who went to selective schools that these data suggest students at such schools study 13 hrs/week, the response is usually something like, "Do you mean per day?"
Finally, since no statistics were run, it's quite possible that none of the results in this study are significant.
I want to be clear that I'm not saying that Babcock's claims aren't true. I'm just saying that it's not clear he has any evidence to support them (which is not to say I think it's a bad study: it was a good study to have done and clearly took a lot of work, but I find it at best suggestive of future avenues of research and certainly not conclusive).
3 comments:
It is unequivocal in the data that students at selective colleges study less they used to. There is no sleight of hand here and it’s worth clearing things up. We compare selective schools in 1961 with *the same set* of highly selective schools in 2003. Study times fell from 26.2 hrs/week to 16.9 hours per week for highly selective colleges, and the decline is highly statistically significant. This is what is graphed in the AEI summary paper. The source paper (Falling Time Cost of College) doesn’t show the breakdown by selectivity between 1961 and 2003 because selectivity is defined there using average SAT scores (which of course were not available in 1961.) But it’s not hard to get around that problem by defining colleges as “selective” based on entering SAT scores in 1981 or later. When it became clear that people wanted to see that breakdown, we included it in the AEI policy paper, along with a few other new results. Again, all of these declines are highly statistically significant and compare identical sets of schools.
While I take your point on apples/oranges comparisons, I guarantee you that the average size of football players at any given institution is up radically from 1930 or, indeed, 1980.
@Babcock -- Thank you for explaining your point. I had difficulty finding it in the manuscript, but perhaps I read too quickly.
I understand your point. However, I'm not sure it addresses the worry that I -- and I think a lot of people -- have. That is, there may still be the same number of students out there studying hard, but the total number of students has increased and so they get swamped out.
Using modern SAT scores may or may not help. The schools may have changed. Certainly, SAT scores have changed, as they must be constantly renormed.
Finally, if you're still around, I'm really concerned about the finding that students at selective schools study less than those at non-selective schools. Should we believe that? What would have to be true about the world for that to be the case? And if we don't believe it, does that make us worry about the data in general? I'm saying that I'm willing to believe this, but I can't take it on face value.
Post a Comment