Who knows more words? Americans, Canadians, the British, or Australians?

I have been hard at work on preliminary analyses of data from the the Vocab Quiz, which is a difficult 32 word vocabulary test. Over 2,000 people from around the world have participated so far, so I was curious to see which of the English-speaking nationalities was doing best.

Since the test was made by an American (me), you might expect Americans to do best (maybe I chose words or definitions of words that are less familiar to those in other countries). Instead, Americans (78.4% correct) are near the bottom of the heap, behind the British (79.8%), New Zealanders (82.2%), the Irish (80.1%), South Africans (83.9%), and Australians (78.6% -- OK that one is close). At least we're beating the Canadians (77.4%).


A fluke?

Maybe that was just bad luck. Plus, some of those samples are small -- there are fewer than 10 folks from New Zealand so far. So I pulled down data from the Mind Reading Quotient, which also includes a (different) vocabulary test. Since the Mind Reading Quotient has been running longer, there are more participants (around 3,000). The situation was no better: This time, we weren't even beating the Canadians. 

Maybe this poor showing was due to immigrants in America who don't know English well? Sorry -- the above results only include people whose native language is English. 

I also considered the possibility  that maybe Americans are performing poorly because I designed the tests to be hard, inadvertently including worse that are rare in America but common elsewhere. But the consistency of results across other countries makes that seem unlikely: What do the British, New Zealanders, Irish, South Africans and Australians all know that we don't? This hypothesis suggests that the poor showing by Americans is due to one or two items in particular. Right now there isn't enough data to do item-by-item analyses, but once we have more. Which brings me to...

Data collection continues

If you want to check how good your vocabulary is compared to everyone else who has taken the test -- and if you haven't done so already -- you can take the Vocab Quiz here. At the Mind Reading Quotient, you can test your ability to understand other people -- to read between the lines.

Update:

Phytophactor asks whether these results are significant. In the MRQ data, all the comparisons are significant, with the exception of US v. Canada (which went the other direction in the Vocab Quiz data anyway). The comparison with Australia is a trend (p=.06). See comments below for additional details. I did not run the stats for Vocab Quiz.

Children don't always learn what you want

Someone has not been watching his/her speech around this little girl.



It's clear she has some sense as to what the phrase means, but clearly she's got the words wrong. But she is treating this phrase as compositional (notice how she switches between "his" and "my").

One of my younger brothers went around for a couple months saying "ship" whenever anything bad happened. But unfortunately we don't have that on video.

Taking research out into the wild

Like others, we believe that science is a little bit WEIRD — much of research is based on a certain type of person, from a very specific social, cultural, and economic background (WEIRD stands for Western Educated Industrialized Rich Democratic; Henrich, Heine, Norenzayan, 2010).  We want to use the web and the help of citizen scientists to start changing that.  In the next few months, we will be launching an initiative called Making Science Less Weird (stay tuned).

As part of Making Science Less Weird, we have proposed a panel presentation at the SXSW conference next year.  Here, "we" includes the team at gameswithwords.org but also at testmybrain.org and labinthewild.org.

In order to be selected, however, *we need votes*. To support Making Science Less Weird and help us increase diversity in human research, please go to this link to create an SXSW account:

https://auth.sxsw.com/users/sign_up

Then go to this link and click on the thumb’s up (on the left under “Cast Your Vote”) to vote for us:

http://panelpicker.sxsw.com/vote/20525

Thanks for your support!

What makes interdisciplinary work difficult

I just read "When physicists do linguistics." Yes, I'm late to the party. In my defense, it only just appeared in my twitter feed. This article by Ben Zimmer describes work published earlier this year, in which a group of physicists applied the mathematics of gas expansion to vocabulary change. This paper was not well received. Among the experts discussed, Josef Fruehwald, a University of Pennsylvania graduate student, compares the physicists to Intro to Linguistics students (not favorably).

Part of the problem is that the physicists seem to have not understood the dataset they were working with and were in any case confused about what a word is, which is a problem if you are studying words! Influential linguist Mark Liberman wrote "The paper's quantitative results clearly will not hold for anything that a linguist, lexicographer, or psychologist would want to call 'words.'"

Zimmer concludes that

Tensions over [the paper] may really boil down to something simple: The need for better communication between disciplines that previously had little to do with each other. As new data models allow mathematicians and physicists to make their own contributions about language, scientific journals need to make sure that their work is on a firm footing by involving linguists in the review process. That way, culturomics can benefit from an older kind of scholarship -- namely, what linguists already know about humans shape words and words shape humans.

Beyond pointing out that linguists and other non-physicists don't already apply sophisticated mathematical models to language -- there are several entire fields that already do this work, such as computational linguistics and natural language processing -- I respectfully suggest that involving linguists at the review process is way too late. If the goal is to improve the quality of the science, bringing in linguists to point out that a project is wrong-headed after the project is already completed doesn't really do anyone much good. I guess it's good not to publish something that is wrong, but it would be even better to publish something that is right. For that, you need to make sure you are doing the right project to begin with.

This brings me to the difficulty with interdisciplinary research. The typical newly-minted professor -- that is, someone just starting to do research on his/her own without regular guidance from a mentor/advisor -- has studied that field for several years as an undergraduate, 5+ years as a graduate student, and several more years as a post-doc. In fact, in some fields even newly-minted professors aren't considered ready to release into the wild and are still working with a mentor. What this tells me is that it takes as much as 10 years of training and guidance before you are ready to be fully on your own. (This will vary somewhat across disciplines.)

Now maybe someone who has already mastered one scientific field can master the second one more quickly. I'm frankly not sure that's true, but it is an empirical question. But it seems very unlikely that anyone, no matter how smart nor how well trained in their first field, is ready to tackle big questions in a new field without at least a few years of training and guidance from an experienced researcher in that field.

This is not a happy conclusion. I'm getting a taste of this now, as I cross-train in computational modeling (my background is pure experimental). It is not fun to go from being regarded as an expert in your field to suddenly being the least knowledgeable person in your laboratory. (After a year of training, it's possible I'm finally a more competent computational modeler than at least the incoming graduate students, though it's a tough call -- they, at least, typically have several years of relevant undergraduate coursework.) And I'm not even moving disciplines, just sub-disciplines within cognitive science!

So it's not surprising that some choose the "shortcut" of reading a few papers, diving in, and hoping for the best, especially since the demands of the career mean that nobody really has time to take a few years off to learn a new discipline. But it's not clear that this is a particularly effective strategy. All the best interdisciplinary work I have seen -- or been involved in -- involved an interdisciplinary team of researchers. This makes sense. It's hard enough to be an expert in one field. Why try to be an expert in two fields when you could just collaborate with someone who has already done the hard work of becoming an expert in that discipline? Just sayin'.

VerbCorner (and others) on SciStarter.Com

There is a brief profile of our crowd-sourcing project VerbCorner on SciStarter.com, with a number of quotes form yours truly.

SciStarter profiles a lot of Citizen Science / Crowd-sourced Science projects. Interestingly, most are physical sciences, with only one project listed under psychology (interestingly, also a language project).

This is not a feature of SciStarter but more a feature of Citizen Science. The Scientific American database only lists two projects under "mind and brain" -- and I'm pretty sure they didn't even have that category last time I checked. This is interesting, because psychologists have been using the Internet to do research for a very long time -- probably longer than anyone else. But we've been very late to the Citizen Science party.

Not, of course, that you shouldn't want to participant in non-cognitive science projects. There are a bunch of great ones. I've personally mostly only done the ones at Zooniverse, but SciStarter lists hundreds.

Peaky performance

Right now there is a giant spike of traffic to GamesWithWords.org, following Steve Pinker's latest tweet about one of the experiments (The Verb Quiz). I looked back over the five years since I started using Google Analytics, and you can see that in general traffic to the site is incredibly peaky.

The three largest single-day peaks account for over 10% of all the visitors to the site over that time period.

Moral of the story: I need Pinker to tweet my site every day!

Findings: GamesWithWords.org at DETEC2013

I recently returned from the inaugural Discourse Expectations: Theoretical, Experimental, and Computational Perspectives workshop, where I presented a talk ("Three myths about implicit causality") which ties together a lot of the pronoun research that I have been doing over the last few years, including results from several GamesWithWords.org experiments (PronounSleuth, That Kind of Person, and Find the Dax).

VerbCorner: New and improved, with surprise bonuses

After a month-long tour, VerbCorner returned to the garage for some fine-tuning. There are now bonus points in each task, doled out whenever ... well, play to find out!

The other major change is that you no longer have to log in to participate. This way, people can check VerbCorner out before committing to filling out the registration form. (Though please do register).

We also made a number of other tweaks here and there to make the site easier to use.

Keeping up to date

Recently, we've added several methods of keeping up to date on GamesWithWords.org projects (finding out when results of old studies are available, when new studies are posted, etc.). In addition to following this blog, that is.

1. Join the GamesWithWords.org Google Group for occasional (5x/year) email updates.

2. Follow @gameswithwords on Twitter.

3. Like our Facebook page.

Citizen Science: Rinse & Repeat

One of the funny things about language is that everybody has their own. There is no "English" out there, existing independently of all its speakers. Instead, there are about one billion people out there, all of whom speak their own idiolect. Most likely, no two people share exactly the same vocabulary (I know some words you might not, possibly including idiolect, and you know some words I don't). Reasonable people can disagree about grammar rules, particularly if one is from Florida and the other from Northern Ireland.

This is one of the reasons we decided to ask people to create usernames in order to contribute to VerbCorner. Suppose two people answer the same question on VerbCorner but disagree. One possibility is that one of them made a mistake (which happens!). But another possibility is that they actually speak different dialects of English, and both are correct (for their dialect). It's hard to tell these possibilities apart by looking at just one question, but by looking at their answers to a set of questions, we can start to get a handle on whether this was a mistake or a real disagreement. The more answers we get from the same person -- particularly across different tasks -- the easier it is to do these analyses.

If we didn't have usernames, it would be hard to figure out which answers all belong to the same person. This is particularly true if the same person comes back to the website from time to time.

People are coming back. At last check, we have ten folks who have answered over 500 questions and four who have answered over 1000. (You can see this by clicking "more" on the leader-board on the main page).

Still, it would be great if we had even more folks who have answered large numbers of questions. Our goal is to have everyone in the top 20 to have answered at least 500 questions by the end of the month.

What makes a sentence ungrammatical?

This is the latest in a series of posts explaining the scientific motivations for the VerbCorner project.


There are many sentences that are grammatical but don't make much sense, including Chomsky's famous “colorless green ideas sleep furiously,” and sentences which seemed perfectly interpretable but are grammatical, such as “John fell the vase” or “Sally laughed Mary” (where the first sentence means that John caused the vase to fall, and the second sentence means that Sally made Mary laugh). You can hit at a window or kick at a window but not shatter at a window or break at a window (unless you are the one shattering or breaking!).

Sentence frames

Notice that these are not agreement errors (“Sally laughed”) or other word-ending errors ("Sally runned to the store"), but instead have something to do with the structure of the sentence as a whole. Linguists often refer to these sentence structures as "frames". There is the transitive frame (NOUN VERB NOUN), the intransitive frame (NOUN VERB), the 'at' frame (NOUN VERB at NOUN), etc. And it seems that certain verbs can go in some frames but not others.

There are many sentence frames (there is disagreement about exactly how to count them, but there are at least a few dozen), and most verbs can appear in somewhere around a half dozen of them. For instance, "thump" can appear in at least eight frames:


NOUN VERB NOUN:                                                  John thumped the door.
NOUN VERB NOUN with NOUN:                             John thumped the door with a stick.
NOUN VERB NOUNs together:                                   John thumped the sticks together.
NOUN VERB NOUN ADJECTIVE:                           John thumped the door open.
NOUN VERB NOUN ADJECTIVE with NOUN:       John thumped the door open with a stick.
NOUN VERB NOUN to [STATE]:                              John thumped the door to pieces.
NOUN VERB NOUN to [STATE] with NOUN:         John thumped the door to pieces with a stick.
NOUN VERB NOUN against NOUN:                         John thumped the stick against the door.

But there are a large number of frames "thump" can't appear in (at least, not without a lot of straining), such as:

NOUN VERB NOUN that SENTENCE:                    John thumped that Mary was angry.
NOUN VERB NOUN NOUN:                                    John thumped Mary the book.
NOUN VERB easily:                                                   Books thump easily.
There VERB NOUN out of [LOCATION]:               There thumped John out of the house.
NOUN VERB what INFINITIVE:                             John thumped what to do.
NOUN VERB INFINITIVE:                                      John thumped to sing


Explaining language

Perhaps these are just funny facts that we must learn about the language we speak, with no rhyme or reason. This is probably true for some aspects of grammar, like which verbs are irregular (that the past tense of “sleep” is “slept” is a historical accident). But a lot of researchers have suspected that there is a reason why language is the way it is and why certain verbs can go into certain frames but not others.

Going back several decades, researchers noticed that when you sort sentences based on the kind of sentence frames they can fit into, you do not get incoherent jumbles of verbs, but rather groups of verbs that all seem to share something in common. So “shatter” and “break” can be used with the object that is shattering or breaking as the direct object ("John shattered/broke the vase") or as the subject ("The vase shattered/broke"). All the verbs that can do this seem to describe some caused change of state (the vase is changing). Verbs that do not describe some kind of caused change cannot appear in both of these forms (you can say “John hit/kicked the vase" but not "The vase hit/kicked" -- at least not without a very special vase!).

Causality might also explain why you can hit at a window or kick at a window but not shatter or break at a window: the addition of the preposition "at" suggests that the action was ineffectual (you tried hitting the window without doing much damage) which is simply nonsensical with words that by their very definition require success. How do you ineffectually shatter a window? You either shatter it or you don't.

So maybe which verbs can go in which frames is not so mysterious after all. Maybe it is a simple function of meaning. Certain verbs have the right meanings for certain sentence frames. No more explanation necessary.

The VerbCorner Contribution


When you group verbs based on the frames they can appear in, you get several hundred groups of verbs in English. Of these, only a handful have been studied in any detail. While it does look like those groups can be explained in terms of their meaning, you might wonder if perhaps these are unusual cases, and if researchers looked at the rest, we would find something different. In fact, a number of researchers have wondered just that.

The difficulty has always been that there are a lot of verbs and a lot of groups. Studying just one group can take a research team years. Studying all of them would take lifetimes.

This is why we decided to crowd-source the problem. Rather than have a few people spend a lifetime, if a lots of people each contribute just a little, we can finish the project in a couple years, if not sooner.

Contribute to the VerbCorner project at gameswithwords.org/VerbCorner/

Bad Evolutionary Arguments

The introductory psychology course I teach for is very heavy on evolutionary psychology. The danger with evolutionary explanations is that it's pretty easy to come up with bad ones. Here's the best illustration I've seen, from Saturday Morning Breakfast Cereal:

How do you tell a good evolutionary argument from a bad one? It's hard to test them with experiments, but that doesn't mean you can't get data. Nice supporting evidence would be finding another species that does the same thing. This hypothesis makes the clear -- and almost certainly false -- prediction that people are likely to adopt babies that fly in out of the blue. You would want to show that whatever reproductive advantage comes from having your genes spread widely (adopted children themselves have more children?) is not overwhelmed by the disadvantages of not being raised by your biological parents (there is data showing that, all else equal, step-parents invest lest in step-children than biological parents invest in their biological children. I expect this generalizes to adoptive parents, but I'm not sure; it might be confounded in modern day by the rigorous screening of adoptive parents).

Etc. We try to teach our students to critically evaluate evolutionary hypotheses. Hopefully it has taken.

Citizen Science Project: Likely Events

VerbCorner was our first step towards opening up the rest of the process. I have just opened up a new good to segment of the website called “Experiment Creator”, which is our second endeavor.

Experiment Creator


One of the most important parts of language experiments is choosing the stimuli. For many types of research, such as in many low level or mid-level vision projects, the experimenter has free reign to design essentially what ever stimuli they like. Language researchers are constrained by the fact that someone suggest other words don't, and each word that has the properties you want may also have other properties that you don't want along for the ride. For instance, you might want to compare nouns and verbs, which don't just differ in terms of part of speech but also frequency (there are many very low-frequency nouns) and length (in some languages, nouns will be systematically longer than verbs; in other languages, it will be the reverse).

Typically, we have to run one or more “norming” experiments to choose stimuli that are controlled for various nuisance factors. These are not really experiments. There is no hypothesis. The purpose of the experiment is indirect (it's an experiment to create another experiment). So I usually do not post them at gameswithwords.org, which recruits people who want to participate in experiments.

The new Experiment Creator project changes this. The tasks posted there will be meta-experiments, used to choose stimuli for other experiments. I just posted the first one, Likely Events.

Likely Events


One of the big discoveries about language in the last few decades is that when we are listening to someone talk or reading a passage, we are actively predicting what will come next. If you hear “John needed money, so he went to the…” you probably expect the next word to be “ATM," not “hibernate.” There are two reasons: 1) "the" is usually followed by a noun, not a verb, and 2) "hibernate" is a relatively rare word.

Much of this research has focused on word frequency and what words follow what other words. We are developing several projects to look more carefully at predictions based not on what word follows what word but on what event is likely to follow what event. In general, "the street" is a more common sequence of words than "the ATM" and "street" is more common than "ATM", but you probably didn't think that the example sentence above was likely to end with "street" for a simple reason: That's not (usually) where you go when you need money.

To do this research, we need to have sequences of events and vary how likely it is that the one event would follow the other, as well as how likely each event is to happen on its own. And we need many, many such sequences. If you would like to help us out, you can do so here.

On the theory that the people interested in these projects will be more committed, Likely Events takes a bit longer than our typical project (in order to make up for the smaller number of volunteers). I expect participation will take on the order of half an hour. We will see how this goes and how many people are interested. Feedback is welcome.

VerbCorner: A Citizen Science project to find out what verbs mean

Earlier this week, I blogged about our new VerbCorner project. At the end, I promised that there would be more info forthcoming about why we are doing this project, about its aims and expected outcomes, why it's necessary, etc. Here's the first installment in that series.

Computers and language

I just dictated the following note to Siri

Many of our best computer systems treat words as essentially meaningless symbols that need to be moved around.

Here's what she wrote

Many of our best computer system street words is essentially meaningless symbols that need to be moved around.

I rest my case.

The problem of meaning.

I don't know for sure how Siri works, but her mistake is emblematic of how much language software works. Computer systems treat and Computer system street sound approximately the same, but that's not something most humans would notice because the first interpretation makes sense and the second one doesn't. 

Decades of research shows that human language comprehension is heavily guided by plausibility: when there are two possible interpretations of what you just heard, go for the one that makes sense. This happens speech recognition like in the example above, and it plays a key role in understanding ambiguous words. If you want to throw Google Translate for a look, give it the following:

John was already in his swimsuit as we reached the watering hole. "I hope the tire swing is still there," John said as he headed to the bank.

Although the most plausible interpretation of bank here is side of a river, Google Translate will translate it into the word for "financial institution" in whatever language you are translating into, because that's the most common meaning of the English work bank.

So what's the problem?

I assume that this limitation is not lost on the people at Google or at Apple. And, in fact, there are computer systems that try to incorporate meaning. The problem there is not so much the computer science as the linguistic science.** Dictionaries notwithstanding, scientists really do not know very much about what words mean, and it is hard to program the computer to know what the word means when you actually do not know.

(Dictionaries are useful, but as an exercise, pick* definition from a dictionary and come up with a counterexample. It is not hard.)

One of the limitations is scope. Language is huge. There are a lot of words. So scientists will work on the meanings of a small number of words. This is helpful, but a computer that only knows a few words is pretty limited. We want to know the meanings of all words.

Solving the problem

We've launched a new section of the website, VerbCorner. There, you can answer questions about what verbs mean. Rather than try to work out the meaning of a word all at once, we have broken up the problem in a series of different questions, each of which tries to pinpoint a specific component of meaning. Of course, there are many nuances to meaning, but research has shown that certain aspects are more important that others, and we will be focusing on those.

I will be writing a lot more about this project, it's goals, the science behind it, and the impact we expect it to have over the coming weeks. In the meantime, please check it out.

----
*Dragon Dictate originally transcribed this as "pickled", which I did not catch on proofreading. More evidence that we need computer programs that understand what words mean.
**Dragon Dictate make spaghetti out of this sentence, too.

Citizen Science at GamesWithWords.org: The VerbCorner Project

What do verbs mean? We'd like to know. For that reason, we just launched VerbCorner, a massive, crowd-sourced investigation into the meanings of verbs. 

Why do we need this project? Why not just look up what verbs mean in a dictionary? While dictionaries are enormously useful (I think I own something like 15), they are far from perfect. For one thing, it's usually very easy to find counter-examples even for what seem like straight-forward definitions. Take the following:

Bachelor: An unmarried man.

So is the Pope a bachelor? Is Neil Patrick Harris? How about a married man from a country in which men are allowed multiple wives?

At VerbCorner, rather than trying to work out the whole definition at once, we have broken meaning into many different components. At the site, you will find several different tasks. In each task, you will try to determine whether a particular verb has a particular component of meaning. 

If you are interested in what words mean and would like to help with this project, sign up for an account at http://gameswithwords.org/VerbCorner/. Participation can be anonymous, but we are happy to recognize significant contributions from anyone who wishes it.

I will be writing a lot more about this project, it's goals, the science behind it, and the impact we expect it to have over the coming weeks. In the meantime, please check it out.