Who knows more words? Americans, Canadians, the British, or Australians?

I have been hard at work on preliminary analyses of data from the the Vocab Quiz, which is a difficult 32 word vocabulary test. Over 2,000 people from around the world have participated so far, so I was curious to see which of the English-speaking nationalities was doing best.

Since the test was made by an American (me), you might expect Americans to do best (maybe I chose words or definitions of words that are less familiar to those in other countries). Instead, Americans (78.4% correct) are near the bottom of the heap, behind the British (79.8%), New Zealanders (82.2%), the Irish (80.1%), South Africans (83.9%), and Australians (78.6% -- OK that one is close). At least we're beating the Canadians (77.4%).


A fluke?

Maybe that was just bad luck. Plus, some of those samples are small -- there are fewer than 10 folks from New Zealand so far. So I pulled down data from the Mind Reading Quotient, which also includes a (different) vocabulary test. Since the Mind Reading Quotient has been running longer, there are more participants (around 3,000). The situation was no better: This time, we weren't even beating the Canadians. 

Maybe this poor showing was due to immigrants in America who don't know English well? Sorry -- the above results only include people whose native language is English. 

I also considered the possibility  that maybe Americans are performing poorly because I designed the tests to be hard, inadvertently including worse that are rare in America but common elsewhere. But the consistency of results across other countries makes that seem unlikely: What do the British, New Zealanders, Irish, South Africans and Australians all know that we don't? This hypothesis suggests that the poor showing by Americans is due to one or two items in particular. Right now there isn't enough data to do item-by-item analyses, but once we have more. Which brings me to...

Data collection continues

If you want to check how good your vocabulary is compared to everyone else who has taken the test -- and if you haven't done so already -- you can take the Vocab Quiz here. At the Mind Reading Quotient, you can test your ability to understand other people -- to read between the lines.

Update:

Phytophactor asks whether these results are significant. In the MRQ data, all the comparisons are significant, with the exception of US v. Canada (which went the other direction in the Vocab Quiz data anyway). The comparison with Australia is a trend (p=.06). See comments below for additional details. I did not run the stats for Vocab Quiz.

Children don't always learn what you want

Someone has not been watching his/her speech around this little girl.



It's clear she has some sense as to what the phrase means, but clearly she's got the words wrong. But she is treating this phrase as compositional (notice how she switches between "his" and "my").

One of my younger brothers went around for a couple months saying "ship" whenever anything bad happened. But unfortunately we don't have that on video.

Taking research out into the wild

Like others, we believe that science is a little bit WEIRD — much of research is based on a certain type of person, from a very specific social, cultural, and economic background (WEIRD stands for Western Educated Industrialized Rich Democratic; Henrich, Heine, Norenzayan, 2010).  We want to use the web and the help of citizen scientists to start changing that.  In the next few months, we will be launching an initiative called Making Science Less Weird (stay tuned).

As part of Making Science Less Weird, we have proposed a panel presentation at the SXSW conference next year.  Here, "we" includes the team at gameswithwords.org but also at testmybrain.org and labinthewild.org.

In order to be selected, however, *we need votes*. To support Making Science Less Weird and help us increase diversity in human research, please go to this link to create an SXSW account:

https://auth.sxsw.com/users/sign_up

Then go to this link and click on the thumb’s up (on the left under “Cast Your Vote”) to vote for us:

http://panelpicker.sxsw.com/vote/20525

Thanks for your support!

What makes interdisciplinary work difficult

I just read "When physicists do linguistics." Yes, I'm late to the party. In my defense, it only just appeared in my twitter feed. This article by Ben Zimmer describes work published earlier this year, in which a group of physicists applied the mathematics of gas expansion to vocabulary change. This paper was not well received. Among the experts discussed, Josef Fruehwald, a University of Pennsylvania graduate student, compares the physicists to Intro to Linguistics students (not favorably).

Part of the problem is that the physicists seem to have not understood the dataset they were working with and were in any case confused about what a word is, which is a problem if you are studying words! Influential linguist Mark Liberman wrote "The paper's quantitative results clearly will not hold for anything that a linguist, lexicographer, or psychologist would want to call 'words.'"

Zimmer concludes that

Tensions over [the paper] may really boil down to something simple: The need for better communication between disciplines that previously had little to do with each other. As new data models allow mathematicians and physicists to make their own contributions about language, scientific journals need to make sure that their work is on a firm footing by involving linguists in the review process. That way, culturomics can benefit from an older kind of scholarship -- namely, what linguists already know about humans shape words and words shape humans.

Beyond pointing out that linguists and other non-physicists don't already apply sophisticated mathematical models to language -- there are several entire fields that already do this work, such as computational linguistics and natural language processing -- I respectfully suggest that involving linguists at the review process is way too late. If the goal is to improve the quality of the science, bringing in linguists to point out that a project is wrong-headed after the project is already completed doesn't really do anyone much good. I guess it's good not to publish something that is wrong, but it would be even better to publish something that is right. For that, you need to make sure you are doing the right project to begin with.

This brings me to the difficulty with interdisciplinary research. The typical newly-minted professor -- that is, someone just starting to do research on his/her own without regular guidance from a mentor/advisor -- has studied that field for several years as an undergraduate, 5+ years as a graduate student, and several more years as a post-doc. In fact, in some fields even newly-minted professors aren't considered ready to release into the wild and are still working with a mentor. What this tells me is that it takes as much as 10 years of training and guidance before you are ready to be fully on your own. (This will vary somewhat across disciplines.)

Now maybe someone who has already mastered one scientific field can master the second one more quickly. I'm frankly not sure that's true, but it is an empirical question. But it seems very unlikely that anyone, no matter how smart nor how well trained in their first field, is ready to tackle big questions in a new field without at least a few years of training and guidance from an experienced researcher in that field.

This is not a happy conclusion. I'm getting a taste of this now, as I cross-train in computational modeling (my background is pure experimental). It is not fun to go from being regarded as an expert in your field to suddenly being the least knowledgeable person in your laboratory. (After a year of training, it's possible I'm finally a more competent computational modeler than at least the incoming graduate students, though it's a tough call -- they, at least, typically have several years of relevant undergraduate coursework.) And I'm not even moving disciplines, just sub-disciplines within cognitive science!

So it's not surprising that some choose the "shortcut" of reading a few papers, diving in, and hoping for the best, especially since the demands of the career mean that nobody really has time to take a few years off to learn a new discipline. But it's not clear that this is a particularly effective strategy. All the best interdisciplinary work I have seen -- or been involved in -- involved an interdisciplinary team of researchers. This makes sense. It's hard enough to be an expert in one field. Why try to be an expert in two fields when you could just collaborate with someone who has already done the hard work of becoming an expert in that discipline? Just sayin'.