Measuring the Quality of a Scientific Paper

"Good" is a notoriously difficult word to define. A pretty common and reasonably uncontroversial definition of a good paper, though, is one that has significantly advanced human knowledge. The question is how do we measure that?

If the paper is in your field, you probably have a sense of the impact. But if it's outside of your field, it becomes trickier. A pretty good proxy is how many times the paper has been cited. Pretty much all the work in the study of language over the last 50 years has bounced off of Chomsky's ideas, and you can see this in the fact that his 1965 book Aspects of the Theory of Syntax has been cited 11,196 times...and that's only one of several very influential books.

It takes a few years for a paper or book to start getting citations, though, because it takes time for people to read it, not to mention the fact that a paper that is printed today was probably written anywhere from 1-3 years ago. So for a new paper, you can estimate how big an impact it will have by looking at the quality of the typical paper published in the journal in which the paper in question was published. This is usually measured by -- you guessed it -- how often papers in that journal are cited.

As a recent article in the Times points out, this gets tricky when you want to compare across fields. In some fields, authors meticulously cite everything that could possibly be relevant (linguistics is a good example). Other fields don't have as strong a citation tradition. Conversely, researchers in some fields traditionally work for years at a time on a project and sum up all their findings every few years in a single, high-impact publication (again, linguistics comes to mind). Other fields are more concerned with quickly disseminating every incremental advance. Then there is simply the size of the field: the more people in your field, the more people can cite you (developmental psychology journals tend to have low impact factors simply because developmental psychology is a very small world).

So by looking at citation rates, you might conclude that molecular biology is the most important field of modern thought, and mathematics and computer science are among the least. I'm a fan of molecular biology, but it's hard not to admit that molecular biology would be impossible without recent advances in mathematics and computer science; the reverse is not true.

Starting Assumptions

The idealized scientist might start by questioning everything and assuming nothing. However, one usually has to make starting assumptions to get things going. For instance, David Hume proved that the notion that science works at all is founded on the un-provable assumption that the future will conform to the past (i.e., if e=mc2 yesterday, it will do so again tomorrow).

Starting assumptions can get a bit less metaphysical though. Here is a very telling line in linguist David Pesetsky's influential Zero Syntax from 1995:

It follows from the hunch just described that hypotheses about language should put as small a burden as possible on the child's linguistic experience and as great a burden as possible on the biologically given system, which we call Universal Grammar (UG). Of course, the role of experience is not zero, or else every detail of language would be fixed along genetic lines. Nonetheless, given that linguistics tries to explain, the null hypothesis should place the role of experience as close to zero as possible.

In contrast, there has been a strong trend in psychology -- and folk science, for that matter -- to assume everything is learned and prove otherwise.

Ultimately, if science proceeds as it should, we'll all converge on the same theory somewhere in the middle. In the meantime, wildly divergent starting assumptions often unfortunately lead to folks simply talking in different languages.

A good example is a recent exchange in Trends in Cognitive Sciences. Waxman and Gelman had recently wrote an article about how children's assumptions about the world (they called these assumptions "theories") guide learning even in infancy. Sloutsky wrote a letter to complain that Waxman and Gelman had failed to explain how those assumptions were learned. Gelman and Waxman responded, in essence: "Who says they're learned?"

All three are intelligent, thoughtful researchers, so at the risk of simplifying the issue, Sloutsky's problem with the "innate theories" theory is that nobody has given a good characterization of how those theories are instantiated in the brain, much less how evolution could have endowed us with those innate theories. Sloutsky assumes learning unless proven otherwise.

However, Waxman and Gelman's problem with Sloutsky is that nobody has a good explanation -- even in theory -- of how you could learn anything without starting with some basic assumptions. At the very least, you need Hume's assumption (the future will conform to the past) to even get learning off the ground.

Both perspectives have their strengths, but both are also fatally flawed (which is not a criticism -- there aren't any perfect theories in cognitive science yet, and likely not in any science). Which flaws bother you the most depends on your starting assumptions.

Are college professors underworked?

According to Dick Morris, I've joined a cushy profession. Professors don't teach very much, which makes college expensive. He argues that by requiring faculty to work harder "approximating the work week the rest of us find normal" and holding down some administrative costs, the tuition can be cut in half!

Comments on The Choice sum up the reaction -- mainly, that strong opinions are easy to have if you have no clue what you are talking about. Most have focused on the ridiculous claim that faculty don't work very hard, presumably due to Morris's odd belief that the only time professors spend working is time spent in the classroom. Morris would presumably cringe at the claim that the only time he spends working is the time he is physically typing out an article.

Well, maybe not Morris. There's no evidence in this article, at least, that he spend any time doing research. But most faculty spend a lot of time doing research, preparing for class, grading, sitting on committees, meeting with students, etc. When I find one who works less than 50 hours a week, I'll ask her secret.

There are also some funny numbers. Morris argues faculty typically teach 5 courses per year, spending 18-20 hours in the classroom per week. If they were to teach 8 courses, they'd spend 24 hours in class per week. Increasing the number of courses by 60% seems to only increase hours by 20%-33%. Sounds like profitability through magical thinking.

There is one point that Morris could have made, though: some universities could be made cheaper by having faculty do no research and less preparation for class. This wouldn't necessarily be an ideal situation, but it would be cheaper. The question is whether it's worth the cost.