Measuring the Quality of a Scientific Paper

"Good" is a notoriously difficult word to define. A pretty common and reasonably uncontroversial definition of a good paper, though, is one that has significantly advanced human knowledge. The question is how do we measure that?

If the paper is in your field, you probably have a sense of the impact. But if it's outside of your field, it becomes trickier. A pretty good proxy is how many times the paper has been cited. Pretty much all the work in the study of language over the last 50 years has bounced off of Chomsky's ideas, and you can see this in the fact that his 1965 book Aspects of the Theory of Syntax has been cited 11,196 times...and that's only one of several very influential books.

It takes a few years for a paper or book to start getting citations, though, because it takes time for people to read it, not to mention the fact that a paper that is printed today was probably written anywhere from 1-3 years ago. So for a new paper, you can estimate how big an impact it will have by looking at the quality of the typical paper published in the journal in which the paper in question was published. This is usually measured by -- you guessed it -- how often papers in that journal are cited.

As a recent article in the Times points out, this gets tricky when you want to compare across fields. In some fields, authors meticulously cite everything that could possibly be relevant (linguistics is a good example). Other fields don't have as strong a citation tradition. Conversely, researchers in some fields traditionally work for years at a time on a project and sum up all their findings every few years in a single, high-impact publication (again, linguistics comes to mind). Other fields are more concerned with quickly disseminating every incremental advance. Then there is simply the size of the field: the more people in your field, the more people can cite you (developmental psychology journals tend to have low impact factors simply because developmental psychology is a very small world).

So by looking at citation rates, you might conclude that molecular biology is the most important field of modern thought, and mathematics and computer science are among the least. I'm a fan of molecular biology, but it's hard not to admit that molecular biology would be impossible without recent advances in mathematics and computer science; the reverse is not true.

Starting Assumptions

The idealized scientist might start by questioning everything and assuming nothing. However, one usually has to make starting assumptions to get things going. For instance, David Hume proved that the notion that science works at all is founded on the un-provable assumption that the future will conform to the past (i.e., if e=mc2 yesterday, it will do so again tomorrow).

Starting assumptions can get a bit less metaphysical though. Here is a very telling line in linguist David Pesetsky's influential Zero Syntax from 1995:

It follows from the hunch just described that hypotheses about language should put as small a burden as possible on the child's linguistic experience and as great a burden as possible on the biologically given system, which we call Universal Grammar (UG). Of course, the role of experience is not zero, or else every detail of language would be fixed along genetic lines. Nonetheless, given that linguistics tries to explain, the null hypothesis should place the role of experience as close to zero as possible.

In contrast, there has been a strong trend in psychology -- and folk science, for that matter -- to assume everything is learned and prove otherwise.

Ultimately, if science proceeds as it should, we'll all converge on the same theory somewhere in the middle. In the meantime, wildly divergent starting assumptions often unfortunately lead to folks simply talking in different languages.

A good example is a recent exchange in Trends in Cognitive Sciences. Waxman and Gelman had recently wrote an article about how children's assumptions about the world (they called these assumptions "theories") guide learning even in infancy. Sloutsky wrote a letter to complain that Waxman and Gelman had failed to explain how those assumptions were learned. Gelman and Waxman responded, in essence: "Who says they're learned?"

All three are intelligent, thoughtful researchers, so at the risk of simplifying the issue, Sloutsky's problem with the "innate theories" theory is that nobody has given a good characterization of how those theories are instantiated in the brain, much less how evolution could have endowed us with those innate theories. Sloutsky assumes learning unless proven otherwise.

However, Waxman and Gelman's problem with Sloutsky is that nobody has a good explanation -- even in theory -- of how you could learn anything without starting with some basic assumptions. At the very least, you need Hume's assumption (the future will conform to the past) to even get learning off the ground.

Both perspectives have their strengths, but both are also fatally flawed (which is not a criticism -- there aren't any perfect theories in cognitive science yet, and likely not in any science). Which flaws bother you the most depends on your starting assumptions.

Are college professors underworked?

According to Dick Morris, I've joined a cushy profession. Professors don't teach very much, which makes college expensive. He argues that by requiring faculty to work harder "approximating the work week the rest of us find normal" and holding down some administrative costs, the tuition can be cut in half!

Comments on The Choice sum up the reaction -- mainly, that strong opinions are easy to have if you have no clue what you are talking about. Most have focused on the ridiculous claim that faculty don't work very hard, presumably due to Morris's odd belief that the only time professors spend working is time spent in the classroom. Morris would presumably cringe at the claim that the only time he spends working is the time he is physically typing out an article.

Well, maybe not Morris. There's no evidence in this article, at least, that he spend any time doing research. But most faculty spend a lot of time doing research, preparing for class, grading, sitting on committees, meeting with students, etc. When I find one who works less than 50 hours a week, I'll ask her secret.

There are also some funny numbers. Morris argues faculty typically teach 5 courses per year, spending 18-20 hours in the classroom per week. If they were to teach 8 courses, they'd spend 24 hours in class per week. Increasing the number of courses by 60% seems to only increase hours by 20%-33%. Sounds like profitability through magical thinking.

There is one point that Morris could have made, though: some universities could be made cheaper by having faculty do no research and less preparation for class. This wouldn't necessarily be an ideal situation, but it would be cheaper. The question is whether it's worth the cost.

And now, on the radio

The radio show I discussed a couple weeks ago finally aired. I would have posted earlier, but I wasn't aware it had happened. My appearance is brief, but it was fun to do. The journalist (Michelle Elliot) does a very nice job of discussing birth order effects, so it is definitely worth listening to.

Coglanglab on Scientific American

I have an article on the Scientific American website (part of the Mind Matters blog) this week. It's on the relationship between language and thought. Check it out.

Speaking Chinese

People often talk about speaking 'Chinese,' as if there were a single language called 'Chinese.' There are a number of related Chinese languages, much as there are a number of related Romance languages.

It turns out the situation is worse than we thought, though. Linguists have been discovering new Chinese languages.

My First Radio Interview

In my brief career as a freelance travel & culture writer, I conducted a number of interviews. I had never been interviewed for anything real prior to just finishing a phone interview with a journalist who is considering writing about my birth order research.

Harvard being Harvard, many of my friends have been interviewed by multiple TV and radio shows, and there are periodically camera crews on my floor. But my lab's research is less media-friendly (no dancing parrots), it's not something we normally deal with.

I admit the experience is somewhat disconcerting. I expect my birth order research to be controversial. And while there is really no point in publishing something that is then ignored, the one advantage of being ignored is nobody's likely to send angry emails, feel I misrepresented their findings, or criticize the methods or conclusions. So while I do seek out publicity for these findings (hence the blog, and also an upcoming article I'm writing for a mainstream science magazine), success in achieving that publicity is at least as worrisome as failure. So we'll see how this goes...

New in Developmental Research

Every year, the Harvard Laboratory for Developmental Studies (of which I am a part) sends out a newsletter to all the parents of the kids have participated in our research studies. For every project conducted in the last year, the lead experimenter (usually a grad student or post-doc) writes up the results in layperson-friendly terms. This year's newsletter was just published. Check it out here.

Ambiguity

I'm preparing a speech for later today on unrecognized ambiguity. Many sentences are ambiguous. Often we don't notice that these sentences are ambiguous, because we know what we intend to say. This probably explains many of the (reportedly) real newspaper headlines I'm using in the talk, most of which are worth reading again even if you already know them:

Ten Commandments: Supreme Court says some OK, some not

Federal agents raid gun shop, find weapons

One-armed man applauds the kindness of strangers

Autos killing 110 a day; let's resolve to do better

Dr. Ruth to talk about sex with newspaper editors

Enraged cow injures farmer with ax

Eye drops off shelf

Iraqi head seeks arms

Juvenile court tries shooting defendant

Killer sentenced to die for second time in 10 years

Kicking baby considered to be healthy

Two soviet ships collide -- one dies

William Kelly was Fed Secretary

Kids make nutritious snacks

Milk drinkers are turning to powder

Does birth order affect who you are friends with? Results from a new study

In 1874, in preparing a demographic survey of English scientists, Francis Galton noticed a funny thing: nearly half of all English scientists were oldest or only sons. In the following 135 years, the notion that your birth order affects personality, intelligence, success in the world, and just about anything and everything else has become a mainstay in popular culture. A search on Amazon turned up over 10,000 books, including such titles as The Birth Order Book: Why You are the Way You Are and The Birth Order Connection: Finding and Keeping the Love of Your Life.

Are Birth Order Effects Real?

It may be surprising, then, that the scientific evidence for birth order effects is heavily controversial. Perhaps more studies that failed to find any effect of birth order as studies that do (see Ernst & Angst's Birth Order: Its Influence on Personality), and many studies that do show such effects are hopelessly confounded. Take Galton's original study. In the 1800s, many scientists were men of independent wealth. Generally, the bulk of inherited wealth went to the oldest son. So there were just more oldest and only sons available to be scientists. There were probably also more younger sons in the military, but this shouldn't be taken as evidence that younger children are more violent.

Modern day family size tends to be related to ethnicity and class. Poor families have more children than rich families, so being a middle child actually correlates with being poor. So if poor educational achievement correlates with being a middle child (which it probably does), that may just be that more middle children go to crumbling schools than do oldest or youngest children.

So if there are no birth order effects, what does that mean? Researchers like Steven Pinker (The Blank Slate) and Judith Rich Harris (The Nurture Assumption: Why Children Turn out the Way They Do) point out that if birth order effects are not real then this calls into question whether your home environment has any effect on how you turned out. If the existence or non-existence of siblings doesn't effect your personality, what chance is there that whether your parents read to you had an effect either? (They both go on to marshall additional evidence that the home environment has no lasting effects.)

Seriously?

Harris and Pinker's claims fly in the fact of much of what we hear. Everybody knows that children who are read to do better in school, right? Children who were beaten by their parents beat their own children. Unfortunately, data like these run up against a very basic problem in science: correlation does not equal cause and effect.

For instance, it is almost certainly the case that the height of the youngest child in the family is correlated with the height of the oldest child of the family. I'm tall and my younger brothers are tall, too. But I can't take credit for making them tall. We happen to share the same growth-related genes.

Similarly, maybe the genes that lead a particular parent to be the sort of parent who reads to her child are also genes that make children do well in school. If this sounds implausible ("genes for reading to one's child?"), consider that there is a personality difference between people who like to read and those who don't. The former might read to their children more. And those children, inheriting the same book-worm genes, do well in school.

(If you don't believe behavior is partly inheritable, The Blank Slate does a good job of summarizing the mountain of research that it is. You might also speculate on why cats act like cats, snakes like snakes, and humans like humans. It can't all be learned. My cats were raised by humans from birth, but they have yet to learn to talk, tie their shoes or clean their own litter box.)

Love, Marriage and Birth Order

One of the difficulties with the existing literature is that it tends to test specific claims about birth order (e.g., older children are more likely to be scientists). So if the experiment fails (the hypothesis is disproven), you only know that your specific idea about birth order is wrong. It doesn't mean that birth order doesn't have some other effect.

In trying to think of a very general test of birth order effects, I realized that a century of research has shown that like marries like. Spouses correlate on just about everything measurable, from physique to personality. Certainly, spouses aren't identical twins, but they tend to be more alike one another than you'd expect if people married random strangers off the street. So if birth order affects personality, and people choose spouses based on having similar personalities, spouses should (on average) have similar birth orders.

In a paper to be published by the Journal of Individual Psychology, my co-authors and I found not only do spouses have similar birth orders, best friends do, too. The effect is not huge, but it probably shouldn't be. Nobody thinks birth order rigidly determines personality, and neither do people rigidly marry others with identical personalities. What we expected was a weak correlation.

Unlike much previous research, this effect can't be due to socio-economic factors. Although middle children are more common in poorer families or in families of certain ethnic groups, there is no ethnic group -- can be no ethnic group -- in which older children are more common than younger children. And yet oldest-oldest and youngest-youngest pairings were more common that oldest-youngest pairings.

Personality or Intelligence?

Afficionados will notice there is another possible explanation. There is now convincing evidence that birth order is related to intelligence. Since spouse's tend to IQs correlate, it could be that IQ drove this effect. Whether this is a counter-hypothesis or not depends on whether you think intelligence is completely unrelated to personality, which seems to be more of a definitional question than a scientific one. But it does raise the question of which aspects of behavior are affected by birth order.

Either way, I think these results maintain some hope for the idea that we are at least partly shaped by how we grew up. It's important to recognize that the effects we found in this study were not large, though of course it would be surprising if it were large. Nobody thinks birth order rigidly determines personality, and neither do people always marry clones of themselves. So the correlation was never likely to be large.

Many Thanks

I want to conclude this post by thanking the participants in this study, many of whom may be among the readers. One of the surveys included in this paper was conducted through the Cognition and Language Laboratory on the Web. And of course, I invite anyone who is interesting in helping out in future research to participate in the new experiments we have running on the site.

The New Cognition and Language Laboratory

The lab's website is in the process of being updated. Everything should be easier to find than it was previously, particularly the results from previous experiments (now labeled the 'findings' page).

Do Americans Value Science? New Numbers

A recent Pew survey finds that more Americans think scientists contribute a lot to society (70%) than do doctors (69%), engineers (64%), the clergy (40%), journalists (38%), artists (31%), lawyers (23%) or business executives (21%). The apparent statistical tie between scientists and doctors may be explained, however, by the fact that many people seem to conflate the two. When asked for an important scientific achievement, about half referred to a biomedical advance.

The survey contains a great deal of information and is worth reading in full. A few other things stand out to me: 49% of scientists, but only 17% of the public, think American science is the best in the world. The objective numbers are that American science is the best in the world. True, this has been rapidly changing, which may explain scientists' pessimism. But why is the public unaware of America's huge historical scientific advantage on the world stage? At the very least, this indicates poor PR on the part of US science.

Time Machines

Semantics Summer Reading List

This summer, I organized two book clubs involving people from people in the Laboratory for Developmental Studies. Many of us have difficulty finding time to read, and the hope was that by forming book clubs there would be peer pressure to read some foundational material. The project has been more or less successful for different people, but at the very least I have managed a lot of reading.

The book clubs were organized around language meaning (some combination of the fields of semantics and pragmatics). Based on mutual interest, we have read or are reading the following books:

Levinson (2000) Presumptive Meanings
Quine (1960) Word and Object
Vygotsky (1934) Language & Thought
Fodor (1975) The Language of Thought
Heim & Kratzer (1998) Semantics in Generative Grammar
Tomasello (2003) Constructing a Language

Keep in mind of course that some obvious books are not on this list because we've already read them (for instance, we read Pinker's Learnability & Cognition and Jackendoff's Semantic Structures last winter) and some are not on the list because we plan to read them in the near future (Pustejovsky's The Generative Lexicon is a popular choice for this coming Fall).

That said, for those of you in the field, if you were to read 6 books on semantics & pragmatics over the summer, what would you read?

All Italians Smoke

Although most behavior experiments are conducted in the lab, it's nice to be reminded occasionally that it's possible to conduct experiments in the human's natural environment...such as a nightclub. Italian scientists studied responses to requests for a cigarette at three nightclubs in central Italy.

That scientists would study Italian's smoking behaviors comes as no surprise to anyone who has been reading semantics recently. It seems that half the example sentences in the papers I read involve some variation on "all Italians smoke."