tag:blogger.com,1999:blog-77017574033645141682024-03-12T21:51:36.973-04:00Games with WordsEdwardhttp://www.blogger.com/profile/04295927435118827266noreply@blogger.comBlogger591125tag:blogger.com,1999:blog-7701757403364514168.post-74400743339599476262016-07-22T12:12:00.002-04:002016-07-22T14:43:34.334-04:00WE MOVED! <span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b>DEAREST READERS, </b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b><br /></b></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b>THANK YOU FOR YOUR SUPPORT THROUGHOUT THE YEARS. WE WILL BE DEPARTING FROM THE FIELD OF SCIENCE NETWORK. </b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b><br /></b></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b>FIND US AT OUR NEW BLOGGING HOME AT <a href="https://blog.gameswithwords.org/" target="_blank"><span style="color: #93c47d;">MEDIUM</span></a>. </b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b><br /></b></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large;"><b>THANK YOU! </b></span>Anonymoushttp://www.blogger.com/profile/08471131142443817485noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-18515491083837571902016-03-24T11:05:00.002-04:002016-03-24T11:05:23.737-04:00Citizen Science Works!Earlier this month, we presented the results of the pilot phase of VerbCorner -- our citizen science project probing the nature of linguistic structure -- at a scientific conference (the <a href="https://sites.google.com/site/eventslangcog2016/home" target="_blank">Workshop on Events in Language and Cognition</a>). You can see the poster describing the work <a href="http://joshuakhartshorne.org/posters/EventsInLanguage2016_VerbCorner.pdf" target="_blank">here</a>. <div>
<br /></div>
<div>
For those who don't know or don't remember, in VerbCorner, we're trying to work out the grammar rules that apply to verbs. Why do you say <i>Agnes looked at the wall</i> but not <i>Agnes saw at the wall</i>? Why do you say <i>Bart filled the glass with water</i> but not <i>Bart poured the glass with water</i>? Many -- but not all -- linguists believe that grammatical idiosyncrasies are explained by the meanings of the verbs, but evidence is sketchy. Volunteers have been visiting our website to help analyze the meanings of verbs so we can find out.</div>
<div>
<br /></div>
<div>
<b>High-Quality Analyses by Volunteers</b></div>
<div>
<b><br /></b></div>
<div>
Our <a href="http://gameswithwords.fieldofscience.com/2013/12/results-round-1-crowdsourcing-structure.html" target="_blank">initial work</a> -- the pilot for the pilot, if you will -- suggested that we could get high-quality analyses from volunteers. But that was based on a very small sample. As of late Feb, over 10,000 volunteers had contributed over 525,000 analyses. In general, the agreement between different volunteers was pretty high -- which is a good sign. Just as importantly, we had a smaller set of 'test' items, for which we knew what professional linguists would say. When we combine the analyses of different volunteers for the same sentence in order to get a 'final answer', the results match the analyses of professional linguists very well. This shows that we can trust these results.</div>
<div>
<br /></div>
<div>
<b>Where Quantity Becomes Quality</b></div>
<div>
<br /></div>
<div>
Just as importantly, we were able to analyze a lot of sentences. In the VerbCorner project, we are trying to determine which sentences have which of a very specific set of aspects of meaning. One aspect is whether the sentence involves something changing physical form (example: <i>Agnes broke the vase</i> as opposed to <i>Agnes touched the vase</i>). Another aspect is whether the sentence involves anything applying physical force to anything else (ex: <i>Agnes pushed Bart</i> as opposed to <i>Agnes looked at Bart</i>). </div>
<div>
<br /></div>
<div>
For purposes of bookkeeping, let's call one aspect of meaning for one sentence an 'item.' After combining across different volunteers, the results were clear enough to definitively code 31,429 items. This makes VerbCorner the largest study of it's kind <i>by far</i>. (A typical study might only look at a few hundred items.) </div>
<div>
<br /></div>
<div>
This quantity makes a big difference. Given how small studies usually are, they can only look at one tiny corner of the language. The problem is that that corner might not be representative. Imagine studying what Americans are like by only surveying people in Brooklyn. This tends to lead to disagreements between different studies; one linguist studies "Brooklyn" and another studies "Omaha", and they come to very different conclusions! Unfortunately, language is so complex and so vast, one person can only analyze one corner. This is why we are recruiting large numbers of volunteers to help!</div>
<div>
<br /></div>
<div>
<b>The results</b></div>
<div>
<b><br /></b></div>
<div>
One major question we had was how much the rules of verb argument structure (that is, the kinds of grammatical rules described above) depend on meaning. Some linguists think they depend entirely on meaning: If you know the meaning of a verb, you know what its grammar will be like. Others think meaning has very little role to play. Most linguists are probably somewhere in the middle.</div>
<div>
<br /></div>
<div>
The results suggest that the first group is right: These rules depend almost entirely on meaning. Or maybe even entirely; it's so close it is hard to tell.</div>
<div>
<br /></div>
<div>
The reason I say "suggest," however, is that while we have the biggest study of its kind, it still only covers about 1% of English. So we've gone from studying Brooklyn to studying all of NYC. It's an improvement, but not yet enough. </div>
<div>
<br /></div>
<div>
This is why I called this first phase a "pilot". We wanted to see if we could get high-quality, clearly-interpretable results from working with volunteers. Many researchers thought this would be impossible. After all, linguists have to go through a lot of schooling to learn how to analyze sentences. But a key finding of the <a href="https://en.wikipedia.org/wiki/Citizen_science" target="_blank">Citizen Science</a> movement is that there are a lot of smart enthusiasts out there who may not be professionals but can very much contribute to science.</div>
<div>
<br /></div>
<div>
<b>The next phase</b></div>
<div>
<b><br /></b></div>
<div>
We have set a goal of reaching 50,000 completed items by July 1st. That will require upping our game and increasing the rate at which we're analyzing items by almost 4x. But the beauty of Citizen Science is that this does not really require that much work on anyone's part. If 3,000 volunteers each spend about one hour contributing to the project, we'll more than hit that goal. So please help out, and please tell your friends. You can contribute <a href="http://gameswithwords.org/VerbCorner/" target="_blank">here</a>.</div>
GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-82994809573259431692016-02-15T08:00:00.000-05:002016-02-15T08:00:00.147-05:00GamesWithWords is hiring<i>It is exciting times at GamesWithWords.org as we settle into our new digs at Boston College. The brick-and-mortar lab now has a name: the Language Learning Laboratory @ Boston College (L3@BC). As we build out the group, expect to see a lot more activity around the site, including new features, projects, etc. Speaking of, we are hiring a research assistant. See the posting below:</i><br />
<br />
The brand-new <a href="http://l3atbc.org/" target="_blank">Language Learning Laboratory at Boston College</a> is recruiting a full-time research assistant. The research assistant will work closely with the PI (Dr. <a href="http://www.joshuakhartshorne.org/" target="_blank">Joshua Hartshorne</a>) and graduate students in the lab. Primary responsibilities will include coordinating the lab's crowdsourcing and citizen science activities. For example, over 10,000 volunteers have contributed over 500,000 linguistic judgments as part of the laboratory's <a href="http://www.gameswithwords.org/" target="_blank">VerbCorner</a> project. The research assistant will help coordinate these volunteers for this and other similar projects. S/he will also manage undergraduate researchers working on these projects and engage in public outreach activities such as blogging or creating educational materials. S/he will assist in data-analysis and have the opportunity to attend and present at major scientific conferences.<br />
<br />
Candidates should have an undergraduate degree in psychology, neuroscience, linguistics, computer science, or a related field (or a good explanation as to why they are qualified anyway). Candidates should also have familiarity with one or more computer programming languages (e.g., Python, R, Matlab, C++) or an exceptional quantitative background (i.e., degree in mathematics). Experience with any of the following would be an added advantage: laboratory research, data analysis, management/supervision, science outreach, journalism, machine learning.<br />
<br />
Review of applications will begin immediately. Start date is flexible but not later than 9/1/2016. Members of groups underrepresented in science are particularly encouraged to apply. International candidates are welcomed but must have an MA or equivalent.<br />
<br />
To apply, send to l3atbc@gmail.com: a CV and a one-page essay explaining why you are interested in the position and how it fits with your past experiences and future goals. Please also arrange for letters of recommendation from 2-3 references to be sent to l3atbc@gmail.com.<br />
<br />
Please be sure that your CV lists your degree(s), major/minor, GPA, any relevant classes (psychology, linguistics, computer science, etc.), programming languages with which you have experience (and the nature of that experience), and any other experiences/qualifications you feel are particularly relevant.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-36267620805481662642015-06-04T14:56:00.004-04:002015-06-04T17:34:23.898-04:00Unrealistic Scientific Optimism<div class="separator" style="clear: both; text-align: left;">
Not enough published studies <a href="https://en.wikipedia.org/wiki/Replication_crisis" target="_blank">replicate</a>. There are <a href="http://pps.sagepub.com/content/7/6/531" target="_blank">a number of reasons</a> for this, but one of the most difficult to address is <a href="https://en.wikipedia.org/wiki/Statistical_power" target="_blank">statistical power</a>. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b>Why is low power a problem?</b></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Suppose you want to test the effect of some training on IQ scores. You test 50 subjects: 25 in your experimental condition and 25 in the control condition. That's a fairly typical size for a psychology study. And you get a significant result. You might be tempted to conclude that your manipulation worked, but it might actually be <a href="http://pps.sagepub.com/content/7/6/543.short" target="_blank"><i>more</i> likely that your results are due to chance or experimenter error</a>. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
It depends on how large of an effect your manipulation ought to have. If the training typically raises people's IQs by 7.5 points, your study would only have had a 41% chance of detecting it (given normal assumptions about normal distributions). A more modest 5 point effect could be detected 20% of the time. You'd need a 14 point effect to have a 90% chance of detecting it.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
For reference, a 14 point effect is enough to move someone of average intelligence into the top fifth of humanity. We are in miracle drug territory. More realistically, your manipulation is unlikely to have nudged IQ by more than a point or two, in which case there's essentially no chance your study could have detected it. So if you did detect something, it's probably an error. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b>Well, how much power do studies have?</b></div>
<div class="separator" style="clear: both; text-align: left;">
<b><br /></b></div>
<div class="separator" style="clear: both; text-align: left;">
Concerns about statistical power (among things) have led some researchers to declare that <a href="https://en.wikipedia.org/wiki/John_Ioannidis" target="_blank">more than half of all published findings are false</a>. Other researchers are more sanguine. In short, if you think that the effects we are studying tend to be pretty large (a standard deviation or more), there is little to worry about. If you think they tend to be small, the situation is dire. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Unfortunately, the only way to accurately determine how large an effect is is to measure it multiple times. Obviously, you can't have done that in advance when running a brand new study. You might be able to guestimate based on the typical effect size in your field. Unfortunately, many fields primarily publish significant results. This introduces a bias, because effect size and significance are correlated. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Suppose we ran the above experiment and the true effect size is 7.5 IQ points. On average, that is what we would find. But of course sometimes we'll run the experiment and the effect will be larger and sometimes it will be smaller, simply due to random chance. By paying attention only to the significant results, we're selectively ignoring those experiments that happened, through no fault of their own, to underestimate the effect. This skews our results, and on average we would report an effect of 11.8 IQ points -- much higher than the truth. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b>Effects aren't as big as you think</b>.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The typical effect reported in the psychology literature is about half a standard deviation, which is why I've been using the example of 7.5 IQ points above. However, because of the bias against publishing null results or replications, this is inflated. That means that the expectations of psychologists are inflated. We are used to seeing effects of half a standard deviation or more. As a result, we are going to chronically underestimate the number of subjects we need to run. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Unfortunately, without widespread publication of null results and replications, we cannot say how badly our perception is distorted, because the degree of distortion depends on how large effects really are. I ran a series of simulations involving a two-condition, between-subjects design to see how bad the distortion might be. In these simulations, I assumed that null results are never reported, which is only a slight exaggeration of the psychology literature.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
In the graph below, the true effect size (measured in standard deviations) is on the X-axis, and the average reported effect size is in the Y-axis. The different lines reflect different numbers of subjects per condition.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrQYmPIBoogPifygMVg97kzlzQBvTqUBOZPrzlqC5txGqL5qz-bqzkZoYFlTTic-hbr85uW3kCtg4QcVoVUjaWyKWcgMBOfIrcd2AkBtGz3ZWCg6XiKp1INnC59hMAPz4ItZO4QoLHbIgu/s1600/sims_nostopping.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrQYmPIBoogPifygMVg97kzlzQBvTqUBOZPrzlqC5txGqL5qz-bqzkZoYFlTTic-hbr85uW3kCtg4QcVoVUjaWyKWcgMBOfIrcd2AkBtGz3ZWCg6XiKp1INnC59hMAPz4ItZO4QoLHbIgu/s320/sims_nostopping.png" width="320" /></a></div>
As you can see, if you have 50 or fewer subjects per condition, you'll hardly ever report an effect size smaller than half a standard deviation, even when the true effect size is one tenth of a standard deviation. This is because reliably detecting an effect of one tenth of a standard deviation requires about 2,000 subjects per condition.<br />
<br />
Even with 1,000 subjects per condition, there is some distortion for effects smaller than one quarter standard deviation.<br />
<br />
<br />
<br />
<br />
Note that these simulations assume that the researcher is correcting for multiple comparisons, isn't <a href="https://en.wikipedia.org/wiki/Data_dredging" target="_blank">p-hacking</a>, etc. The situation gets worse if we relax those assumptions.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9kK2O5Zm2vxYrIbbf6HoeTXE4t5zkfHgN7gbUp0W4s2CTKHOfchvhEvolYlTXsoZz0Vyy01ufGKm8MuO1itWSZdEsq_aCmeR37a9TDlxfonAUlUvFfy8jW9RTbCbgvUOjscPnD0cQMA1Z/s1600/sims_stopping10+copy.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9kK2O5Zm2vxYrIbbf6HoeTXE4t5zkfHgN7gbUp0W4s2CTKHOfchvhEvolYlTXsoZz0Vyy01ufGKm8MuO1itWSZdEsq_aCmeR37a9TDlxfonAUlUvFfy8jW9RTbCbgvUOjscPnD0cQMA1Z/s320/sims_stopping10+copy.png" width="320" /></a></div>
In the simulation on the left, I assumed the researcher is engaging in contingent stopping. After every 5 subjects, the researcher checks her data. If the effect is significant, she stops and reports the result. If the effect is in the "wrong direction", she decides there's something wrong with her stimuli, revamps the experiment, and tries again.<br />
<br />
While this is a little extreme, most researchers engage in some amount of contingent stopping. As you can see, this badly distorts the results. Even with 1,000 subjects, we end up distorting even large effects.<br />
<br />
Failure to correct for multiple comparisons will have similar effects.<br />
<br />
What this means is that if you are basing your expected effect size on prior experience, the effect you are trying to detect is probably not anywhere near as large as you think, and you may need many more subjects to detect it than you were planning.<br />
<br />
<b>But I usually get significant results. Doesn't that mean I have enough subjects?</b><br />
<b><br /></b>
Maybe. You might be testing an abnormally large effect. Such effects <a href="https://en.wikipedia.org/wiki/Stroop_effect" target="_blank">do exist</a>. Alternatively, you may be engaging in contingent stopping, failing to adequately correct for multiple comparisons, or simply making a mistake somewhere in your analysis. It's probably worth checking.<br />
<br />
<a href="http://www.ellenrooneydesign.com/gsproduction/sites/default/files/images/DSC_5304mr.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://www.ellenrooneydesign.com/gsproduction/sites/default/files/images/DSC_5304mr.jpg" height="133" width="200" /></a>To adapt <a href="http://pss.sagepub.com/content/early/2015/03/30/0956797614567341.abstract" target="_blank">a metaphor from Uri Simonsohn</a>, you can search for exoplanets with binoculars. But you <br />
should keep in mind that it is so unlikely you could see an exoplanet with your binoculars, that if you do see one, you are probably mistaken in some way.<br />
<br />
<b>I don't use t-tests. How does this apply to me?</b><br />
<b><br /></b>
My examples above use t-tests because they are simple and widely known. But the same basic issues apply no matter what kind of analysis you do. If you are looking for some kind of effect, and if that effect is small, you'll need a lot of data to detect it. And it's probably smaller than you think.<br />
<br />
If you are model-fitting, the noise in your data puts an upper limit on how well you can fit the underlying phenomenon. If your data are fairly noisy and yet your model fits really well, you are probably fitting the noise, not the signal. And your data are probably noisier than you think.<br />
<br />
<b>Recommendations</b><br />
<b><br /></b>
I can't tell you how many subjects to run. I don't know. Nobody knows. Until there is a good mechanism for reporting null results, nobody will know.<br />
<br />
In the meantime, I recommend running more than you think you need. If there are published studies looking at a similar phenomenon, look at their effect size and assume the true effect size is significantly smaller, then do a power analysis. If you run your study and get a positive result, it's probably a good idea to replicate it. If you get a null result, you might not wish to read too much into that.<br />
<br />
If you don't have the means to test a lot of subjects, you have a few options. Effect size is a combination of the signal-to-noise ratio and the amount of data you collected. If you can't collect more data, you can try to decrease the noise (e.g., by using a better, more accurate instrument). You can't necessarily increase the signal, because you can't change the laws of nature. But you can decide which laws of nature to study, you and you might be better off studying one that has powerful effects.<br />
<br />
<b>R code</b><br />
<b><br /></b>
See below for the R code to run the simulations above. If anyone knows how to convince blogspot to allow indention, please lemmeno.<br />
<br />
<div class="p1">
runsim<span class="s1"> <- span=""><span class="s2">function</span><span class="s1"> (</span>N<span class="s1">,</span>d<span class="s1">,</span>iter<span class="s1">,</span>inter<span class="s1">=</span>N<span class="s1">,</span>verbose<span class="s1">=</span><span class="s2">FALSE</span><span class="s1">){</span></-></span></div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span>#N = number of subjects/condition</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span>#d = effect size in Cohen's d</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span>#iter = number of simulations to run</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span>#inter = after this number of subjects, check results. Stop if significant *or* in the wrong direction. Default to running all subjects.</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span>#verbose = should we print interim results?</div>
<div class="p3">
<span class="Apple-tab-span"> </span></div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span></span><span class="s3">ES</span><span class="s1"><-c span="">#vector of effect sizes</-c></span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="s2">for</span> (<span class="s3">i</span> <span class="s2">in</span> <span class="s4">1</span>:<span class="s3">iter</span>){</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>#do iter simulations</div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s2">if</span> (<span class="s3">verbose</span>){print(paste(<span class="s5">"Starting simulation"</span>,<span class="s3">i</span>))}</div>
<div class="p3">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span><span class="s3">con1</span><span class="s1"><- span=""><span class="s3">con2</span><span class="s1"><-c span="">#initialize scores for the two conditions</-c></span></-></span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s2">for</span> (<span class="s3">j</span> <span class="s2">in</span> <span class="s4">1</span>:ceiling(<span class="s3">N</span>/<span class="s3">inter</span>)){</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>#on each round, test an addition inter subjects.</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span><span class="s3">n</span><span class="s1"><- span=""><span class="s3">inter</span><span class="s1"> </span>#how many subjects do we need for this round?</-></span></div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span><span class="s2">if</span><span class="s1"> (</span><span class="s3">j</span><span class="s1">*</span><span class="s3">inter</span><span class="s1">></span><span class="s3">N</span><span class="s1">){</span><span class="s3">n</span><span class="s1"><- span=""><span class="s3">N</span><span class="s1">-</span><span class="s3">j</span><span class="s1">*</span><span class="s3">inter</span><span class="s1">} </span>#adjust if last round would result in more than N subjects</-></span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s3">con1</span><-c class="s3" span="">con1</-c></div>
,rnorm(<span class="s3">n</span>,<span class="s4">0</span>,<span class="s4">1</span>))<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s3">con2</span><-c class="s3" span="">con2</-c></div>
,rnorm(<span class="s3">n</span>,<span class="s3">d</span>,<span class="s4">1</span>))<div class="p3">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s2">if</span> (<span class="s3">verbose</span>){print(paste(<span class="s5">"pval:"</span>,round(t.test(<span class="s3">con1</span>,<span class="s3">con2</span>)$<span class="s3">p.value</span>,<span class="s4">2</span>),<span class="s5">" diff in means:"</span>,round(mean(<span class="s3">con2</span>)-mean(<span class="s3">con1</span>),<span class="s4">2</span>)))}</div>
<div class="p3">
<br /></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s2">if</span> (t.test(<span class="s3">con1</span>,<span class="s3">con2</span>)$<span class="s3">p.value</span><<span class="s3">.05</span>){</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>#reached a significant result!</div>
<div class="p5">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>break</div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s2">if</span> (mean(<span class="s3">con1</span>)>mean(<span class="s3">con2</span>)){</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>#preliminary results are in the wrong direction. Revamp experiment and try again.</div>
<div class="p5">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>break</div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</div>
<div class="p3">
<br /></div>
<div class="p1">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span><span class="s2">if</span><span class="s1"> (t.test(</span>con1<span class="s1">,</span>con2<span class="s1">)$</span>p.value<span class="s1"><</span>.05<span class="s1">){</span></div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="s3">ES</span><-c class="s3" span="">ES</-c></div>
,round((mean(<span class="s3">con2</span>)-mean(<span class="s3">con1</span>))/((sd(<span class="s3">con1</span>)+sd(<span class="s3">con2</span>))/<span class="s4">2</span>),<span class="s4">2</span>))<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}<span class="s2">else</span>{</div>
<div class="p2">
<span class="s1"><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span></span>#wasn't significant, so don't report</div>
<div class="p4">
<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>}</div>
<div class="p4">
<span class="Apple-tab-span"> </span>}</div>
<div class="p4">
<span class="Apple-tab-span"> </span>return(<span class="s3">ES</span>)</div>
<div class="p4">
}</div>
<div class="p3">
<br /></div>
<div class="p2">
## graph observed effect size against actual effect size for different N</div>
<div class="p3">
<br /></div>
<div class="p4">
<span class="s3">results</span><-data .frame="" class="s3" span="">N</-data></div>
=c(rep(<span class="s4">10</span>,<span class="s4">10</span>),rep(<span class="s4">25</span>,<span class="s4">10</span>),rep(<span class="s4">50</span>,<span class="s4">10</span>),rep(<span class="s4">100</span>,<span class="s4">10</span>),rep(<span class="s4">1000</span>,<span class="s4">10</span>)),<span class="s3">GT</span>=c(rep(seq(<span class="s3">.1</span>,<span class="s4">1</span>,<span class="s3">.1</span>),<span class="s4">5</span>)),<span class="s3">ES</span>=<span class="s4">0</span>)<div class="p4">
<br /></div>
<div class="p4">
<span class="s2">for</span> (<span class="s3">i</span> <span class="s2">in</span> <span class="s4">1</span>:length(<span class="s3">results</span>$<span class="s3">N</span>)){</div>
<div class="p1">
results<span class="s1">$</span>ES<span class="s1">[</span>i<span class="s1">]<-mean runsim="" span="">results<span class="s1">$</span>N<span class="s1">[</span>i<span class="s1">],</span>results<span class="s1">$</span>GT<span class="s1">[</span>i<span class="s1">],</span><span class="s4">1000</span><span class="s1">))</span></-mean></span></div>
<div class="p4">
}</div>
<div class="p4">
<br /></div>
<div class="p1">
results<span class="s1">$</span>N<span class="s1"><-as .factor="" span="">results<span class="s1">$</span>N<span class="s1">)</span></-as></span></div>
<div class="p1">
results<span class="s1">$</span>expected<span class="s1"><- span="">results<span class="s1">$</span>GT</-></span></div>
<div class="p4">
<br /></div>
<div class="p4">
library(<span class="s3">ggplot2</span>)</div>
<div class="p4">
quartz(<span class="s3">width</span>=<span class="s4">5</span>,<span class="s3">height</span>=<span class="s4">4</span>)</div>
<div class="p1">
p<span class="s1"> <- ggplot="" span="">data<span class="s1">=</span>results<span class="s1">, aes(</span>x<span class="s1">=</span>GT<span class="s1">, </span>y<span class="s1">=</span>ES<span class="s1">, </span>fill<span class="s1">=</span>N<span class="s1">, </span>linetype<span class="s1">=</span>N<span class="s1">, </span>colour<span class="s1">=</span>N<span class="s1">)) </span></-></span></div>
<div class="p4">
<span class="s3">p</span> <- class="s3" span="">p</-></div>
+ geom_line()+ scale_color_manual(<span class="s3">values</span>=c(<span class="s5">"red"</span>,<span class="s5">"blue"</span>,<span class="s5">"dark green"</span>,<span class="s5">"purple"</span>,<span class="s5">"black"</span>))<div class="p4">
<span class="s3">p </span><- class="s3" span="">p</-></div>
+ scale_x_continuous(<span class="s3">limits</span>=c(<span class="s4">0</span>,<span class="s4">1</span>),<span class="s3">breaks</span>=seq(<span class="s4">0</span>,<span class="s4">1</span>,<span class="s3">.25</span>)) + scale_y_continuous(<span class="s3">limits</span>=c(<span class="s4">0</span>,<span class="s4">1.5</span>),<span class="s3">breaks</span>=seq(<span class="s4">0</span>,<span class="s4">1.5</span>,<span class="s3">.25</span>))
<br />
<div class="p1">
p</div>
GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-56537647452129559262015-06-03T14:21:00.001-04:002015-06-03T14:21:17.863-04:00VerbCorner VideoIn case you wanted to know more about our VerbCorner project.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/On26KJXgK6A" width="560"></iframe>
<br />
Many thanks to the two undergraduates who helped make this video.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-77064771039024800792015-06-03T10:58:00.001-04:002015-06-03T10:58:53.365-04:00Back onlineIt's been a very busy time at GamesWithWords. I'm pleased to announce that we'll be moving to Boston College in January. The impending move, combined with a large number of papers to write, has kept me too busy to write much on this blog.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com2tag:blogger.com,1999:blog-7701757403364514168.post-48118617666791360792014-06-19T19:53:00.001-04:002014-06-19T19:53:21.116-04:00Magic SinglishA number of non-native English speakers get "Singaporean" as the top guess for their native language. You can actually see that by playing around in our <a href="http://www.gameswithwords.org/WhichEnglish/dialect_results.html" target="_blank">dialect navigator</a>. Here's screenshot of a particularly illuminating view:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZNQTZgkeKC75-CByIEp0tZwFjMpo_QHhdZuSh8rFoVQC92v2ecATbtfaAx0waVmZlPF71e7eeFi2e7yA9I1jNgOFul8qLVxXsmIcCX0wd6EI68LGs9j_CeTGhLSXPQ0tOH6pdStlUHA-i/s1600/Screen+Shot+2014-06-19+at+7.33.33+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZNQTZgkeKC75-CByIEp0tZwFjMpo_QHhdZuSh8rFoVQC92v2ecATbtfaAx0waVmZlPF71e7eeFi2e7yA9I1jNgOFul8qLVxXsmIcCX0wd6EI68LGs9j_CeTGhLSXPQ0tOH6pdStlUHA-i/s1600/Screen+Shot+2014-06-19+at+7.33.33+PM.png" height="547" width="640" /></a></div>
<br />
<br />
As you can see, "Singaporean" is connected to a big bundle of non-native dialects. Most of the other native dialects are off in a chain in the bottom right. Here is another view with a slightly weaker filter on connectedness:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja26gDwGEkNTMX5Z2TlStzf_sudd6HqUJqiUK1cnWF5mqVNb9-hoBr_BxXs7monPVaD2VMxWbmiWvZcRKgH0Thb2Ie7-lTXShCnQnd0lhBV03myCzCx2gojM2X_DNeRorFtKYnxLPsv2HS/s1600/Screen+Shot+2014-06-19+at+7.39.47+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja26gDwGEkNTMX5Z2TlStzf_sudd6HqUJqiUK1cnWF5mqVNb9-hoBr_BxXs7monPVaD2VMxWbmiWvZcRKgH0Thb2Ie7-lTXShCnQnd0lhBV03myCzCx2gojM2X_DNeRorFtKYnxLPsv2HS/s1600/Screen+Shot+2014-06-19+at+7.39.47+PM.png" height="554" width="640" /></a></div>
<br />
Again, you can see that most of the non-native dialects cluster together. Most of the native dialects do not connect directly to that cluster but rather connect to Singaporean. Again, you can see Standard American and AAVE off in their own cluster.<br />
<br />
Of course, this view just tells you what is connected to what. It's possible that Swedish is actually more similar to Irish than to Singaporean, even though the chain of connections is farther for Swedish and Irish. If you click on one of the dialects, the panel on the left will show you how closely related that dialect is to all others:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigTPwmBF7K937TfVyzPsseeTEWtRsAvcWajkWtUJSfx3aih9E7ErFdkU5YnZ2ehrKk55A_qGNANKgDGsJwZwA3QI4DDzCKv2kHVeb55XuCgYL9cRhu8oPBxYeMDBhqvYxS-ALsdJBRBenG/s1600/Screen+Shot+2014-06-19+at+7.45.05+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigTPwmBF7K937TfVyzPsseeTEWtRsAvcWajkWtUJSfx3aih9E7ErFdkU5YnZ2ehrKk55A_qGNANKgDGsJwZwA3QI4DDzCKv2kHVeb55XuCgYL9cRhu8oPBxYeMDBhqvYxS-ALsdJBRBenG/s1600/Screen+Shot+2014-06-19+at+7.45.05+PM.png" height="640" width="241" /></a></div>
<br />
We're working on a browser that will let you see *why* different dialects are more or less related -- that is, what answers in the quiz are typical of which dialects. I'm hoping it will be ready soon. In the meantime, enjoy the dialect browser.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com3tag:blogger.com,1999:blog-7701757403364514168.post-48554210310267306272014-06-05T19:22:00.001-04:002014-06-05T19:22:10.275-04:00Updated results on the relationship between English dialectsI've updated the interactive visualization of the relationships between the Englishes of the world to include a couple dozen additional native languages. <a href="http://www.gameswithwords.org/WhichEnglish/dialect_results.html" target="_blank">Check it out</a>.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com1tag:blogger.com,1999:blog-7701757403364514168.post-85112347632476266112014-06-04T15:51:00.001-04:002014-06-04T15:51:41.091-04:00Forums find GamesWithWordsA number of forums have picked up the WhichEnglish quiz, and have produced some really intelligent and insightful conversation. I recommend in particular this conversation on <a href="http://www.metafilter.com/139606/My-first-language-is-Norwegian" target="_blank">metafilter</a>. There is also an extensive conversation at <a href="https://news.ycombinator.com/item?id=7843693" target="_blank">hacker news</a> and a somewhat older discussion at <a href="http://www.reddit.com/r/languagelearning/comments/26ofs2/which_english_do_you_speak_quiz_that_asks_you/" target="_blank">reddit</a>. And there is a lot of discussion in Finnish and Hungarian, but I have no idea what they are saying...GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-73551793901733180572014-06-02T15:00:00.000-04:002014-06-02T15:00:02.028-04:00Handling viral trafficAround 4am EST on May 28, we started getting *a lot* of traffic to the website. This very quickly <a href="http://gameswithwords.fieldofscience.com/2014/05/good-problems-to-have.html" target="_blank">overloaded the server</a>, resulting in the website running very slowly. We did some optimization. Things sped up, and our reward was more traffic. So we switched to a more powerful server. And so on.<br />
<br />
Things are finally under control. At least for the moment, anyway. You can see that we've managed to get the average page load time down to a reasonable length of time for the last day or so, without any large spikes:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSvVxbTBU3aj9RmkHbJ73IBteUmvZTnkdIALvG13CeIPTh0JfHqIIzx7O9XxqnIWjjRaYJHyGhEzoH-yYe2oXmYI6kmggvMIxl7mMNdnjhCgMNGz9pMLVyMcM8uMXIa-g3bTeMUncULXbD/s1600/Screen+Shot+2014-06-02+at+11.29.52+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSvVxbTBU3aj9RmkHbJ73IBteUmvZTnkdIALvG13CeIPTh0JfHqIIzx7O9XxqnIWjjRaYJHyGhEzoH-yYe2oXmYI6kmggvMIxl7mMNdnjhCgMNGz9pMLVyMcM8uMXIa-g3bTeMUncULXbD/s1600/Screen+Shot+2014-06-02+at+11.29.52+AM.png" height="337" width="640" /></a></div>
<br />
<br />
Of course, overwhelming amounts of traffic is a good problem to have, and I won't complain if things overheat again.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com5tag:blogger.com,1999:blog-7701757403364514168.post-59776439019914188912014-06-02T04:17:00.001-04:002014-06-03T10:24:13.489-04:00Which English: The Science, Part 1I've gotten a number of questions about the science behind our <a href="http://www.gameswithwords.org/WhichEnglish/" target="_blank">WhichEnglish</a> quiz. Actually, I had intended to post a more detailed discussion days ago, but I <a href="http://gameswithwords.fieldofscience.com/2014/05/good-problems-to-have.html" target="_blank">got distracted by other matters</a>.<br />
<br />
In this project, we are looking at three interrelated issues:<br />
<br />
1. How does the age at which you start learning a language affect how well you learn that language?<br />
2. How is learning a foreign language affected by the language you already know?<br />
3. How are the grammars of different English dialects related?<br />
<br />
And of course, we train an algorithm to predict participants' native language and dialect of English based on their answers. I return to that at the end.<br />
<br />
<b>Age of Acquisition</b><br />
<br />
Although WhichEnglish has a few scientific targets, age-of-acquisition effects were the original inspiration. Everybody knows that the older you are when you start learning a foreign language, the harder it is to learn. One possibility is that there is a <a href="http://en.wikipedia.org/wiki/Critical_period" target="_blank">critical period</a>: Up to some age, you can learn a language like a native. After that age, you will never learn it perfectly. The other possibility is that there is no specific period for language learning; rather, language-learning simply gets a little harder every day.<br />
<br />
The evidence is unclear. Ideally, you would compare people who started learning some language (say, English) from birth with people who started as 1 year-olds and people who started as 2 year-olds, etc. Or maybe you would want something even finer-grained. The problem is that you need a decent number of people at each age (50 would be a good number), and it quickly becomes infeasible.<br />
<br />
<a href="http://crl.ucsd.edu/courses/hdp1/Readings/Hakuta%20Bialystok%20Wiley.pdf" target="_blank">One study</a> that came close to this ideal used census data. The authors -- led by Kenji Hakuta -- realized that the US census asks foreign-born residents to rate their own English ability. The authors compared this measure of English ability with the year of immigration (an approximation for the age at which the person started learning English). Their results showed a steady decline, rather than a critical period.<br />
<br />
We are trying to build on this work in a few ways. For one, it would be nice to confirm (or disconfirm) the previous results with a more sensitive measure of English ability. So rather than just ask people how good their English is, we have them take a test. Also, we are getting more precise information about when the participant started learning English and in what contexts.<br />
<br />
Also, there is good reason to suspect that age-of-acquisition affects different aspects of language differently. Studies have shown that even people who began learning a language as toddlers have detectable -- if very subtle -- accents. However, people who start learning foreign languages as adults usually report that learning vocabulary isn't so hard. Grammar seems to be somewhere in between. The Hakuta study didn't distinguish these different aspects of language.<br />
<br />
WhichEnglish focuses on grammar. We also have a <a href="http://www.gameswithwords.org/VocabQuiz/" target="_blank">vocabulary quiz</a> to look at vocabulary. A pronunciation test is in the works.<br />
<br />
<b>First language effects</b><br />
<b><br /></b>
When we started thinking about studying age-of-acquisition effects, we quickly realized a problem. We needed questions that would be difficult for someone who learned English as a second language. But which aspects of grammar are difficult seems to depend on your first language. I personally have difficulty with <a href="http://en.wikibooks.org/wiki/Russian/Verbal_Aspect" target="_blank">aspect in Russian</a> because the English aspect system is much less complex. However, dealing with tense in Russian is relatively straightforward, since the Russian tense system is much less complex that English's.<br />
<br />
Since we didn't know for sure what the language backgrounds of our participants would be, we wanted a range of questions that covered the different kinds of problems people with different backgrounds might have.<br />
<br />
As we combed the literature, we realized that it was pretty fragmented. One study might say that grammar rule x is difficult for Japanese-speakers and grammar rule y is difficult for German-speakers, but there would be no information on how Japanese-speakers fare with grammar rule y and how German-speakers manage with grammar rule x. This actually makes sense: Most studies look at speakers of one or at most a handful of language backgrounds. This is partly a matter of research interest (the researchers are usually interested in some particular language) and partly a matter of feasibility (in a lab setting, you can only test so many participants). We realized that our study, by virtue of being on the Internet and recruiting people from a wide array of backgrounds, would provide an opportunity to get more systematic data across a large number of languages.<br />
<br />
This is pretty exploratory. We don't have strong hypotheses. But as data comes in, we will be analyzing to see what we get, and we will report it here.<br />
<br />
<b>The Grammars of English</b><br />
<br />
In designing our age-of-acquisition study, we realized a second problem. Correct English grammar varies across different dialects. In Newfoundland, you can say "Throw me down the stairs the hammer," but most places, you can't. (I have heard that this is said in parts of Rhode Island, too, but only anecdotally.) We don't want to count a late-learner of English who says "Throw me down the stairs the hammer" as not knowing English if in fact she lives in Newfoundland!<br />
<br />
So what we really wanted were questions for which the correct answer is the same in all English dialects. But we didn't know what those were. Again, the literature was only partly helpful here. For obvious reasons, researchers tend to be interested in understanding peculiar constructions specific to certain dialects, rather than recording what is the same everywhere (boring).<br />
<br />
We picked out a lot of grammar rules that we at least had no reason to believe varied across dialect. But we also realized that there was an opportunity here to study differences across dialects. So we included a subset of items that we thought probably would be different across dialects so that we can explore relationships across dialects.<br />
<br />
<b>The algorithm</b><br />
<b><br /></b>
When you take the quiz, at the end we give you our best guess as to what your native language is and what dialect of English you speak. How is that related to the three issues I just discussed?<br />
<br />
It's deeply related. The best way of proving that you understand how people's understanding of grammar is affected by the age at which they started learning, their first language (if any), and the dialect of English they speak, is to show that you can actually distinguish people based on their grammar. In fact, training an algorithm to make just that distinction is a common way of analyzing and exploring data.<br />
<br />
There are also obvious practical applications for an algorithm that can guess someone's language background based on their grammar (for education, localization of websites, and so on).<br />
<br />
But an important reason we included the algorithm's predictions in the quiz itself was to present the results of the study to participants in the study as the study goes on. Certainly, you can read this and <a href="http://gameswithwords.fieldofscience.com/search/label/WhichEnglish" target="_blank">other blog posts</a> I've written about the project as well. But it probably took you as long to read this post as to do the quiz. The algorithm and its predictions boil down the essence of the study in a compelling way. Based on the (numerous) emails I have gotten, it has inspired a lot of people to think more about language. Which is great. The best Web-based studies are a two-way street, where the participants get something out of the experience, too.<br />
<br />
We chose the particular algorithm we use because it runs quickly and could be trained on very little data. You can read more about it by clicking on "how it works" in our <a href="http://www.gameswithwords.org/WhichEnglish/dialect_results.html" target="_blank">data visualization</a>. We are testing out more sophisticated algorithms as well, which are likely to do much better. Algorithms for detecting underlying patterns is actually a specialty of my laboratory, and this will be a fantastic dataset to work with. These algorithms mostly run too slowly to use as part of the quiz (nobody wants to wait 10 minutes for their results), but the plan is to describe those results in future posts and/or in future data visualizations.<br />
<br />
<b>In conclusion</b><br />
<b><br /></b>
If you have any questions about this work, please ask in the comments below or shoot me an email at gameswithwords@gmail.com.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com34tag:blogger.com,1999:blog-7701757403364514168.post-48155657268676525962014-05-29T00:35:00.001-04:002014-06-02T01:50:56.818-04:00Good problems to haveGamesWithWords.org will be experiencing periodic outages as we upgrade* the server. The incredible response we've had for <a href="http://www.gameswithwords.org/WhichEnglish/" target="_blank">WhichEnglish</a> has completely overwhelmed the server. After bringing it back from the dead multiple times, the techs at <a href="http://datarealm.com/" target="_blank">Datarealm</a> convinced me to upgrade to the next tier of server.<br />
<br />
This is possibly overkill, in that we don't normally get the kind of traffic we got today. Over 12% of *all* visitors to the website since Jan. 1, 2008, came in the last 24 hours! Still, traffic has been steadily rising over the last year, and large spikes are getting much more frequent.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYCPnb5xzlT7EhjqklQPYw15C4rS0MLNR7txorpJ04GUVFuhLaV8Q10D5JIoLq9PDStWTMYIBtGOcHWXRTGa_AjlG2XgH5sP7gtxXbFuydNtZy5tF4P7axxi1nPLA_hk03Kka7aTMzmp1t/s1600/TheSpike.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYCPnb5xzlT7EhjqklQPYw15C4rS0MLNR7txorpJ04GUVFuhLaV8Q10D5JIoLq9PDStWTMYIBtGOcHWXRTGa_AjlG2XgH5sP7gtxXbFuydNtZy5tF4P7axxi1nPLA_hk03Kka7aTMzmp1t/s1600/TheSpike.png" height="262" width="640" /></a></div>
<br />
<br />
Worst case scenario, this should result in a faster, more stable experience for people going forward.<br />
<br />
*Upgrading while there is heavy traffic to your website is not ideal. But then neither is having the site crash constantly.<br />
<br />
<b>**Update**</b><br />
<br />
After my optimistic comments about "overkill", I've spent most of the last 5 days performing various upgrades to the server. Traffic to the site peaked at about 100,000 visits/day (it was a little lower Sunday, but then weekend traffic is usually down).<br />
<br />
There was a lot I could do to shrink page-load time (compressing images, minimizing javascript files, etc.). But the biggest issues were with sending data to and from the database. Here, I did some work to optimize and cut down the number of calls to the database, but the real heroes are the folks at <a href="http://www.datarealm.com/" target="_blank">Datarealm</a>, who -- based on the amount of time they've put into helping me with the site over the last week -- have definitely lost money on having me as a client. If you are looking for someone to host your website, I warmly recommend them.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-50572058911792078162014-05-23T00:47:00.000-04:002014-05-26T17:35:14.758-04:00Findings: Which English -- updated dialect chartI have updated the dialect chart based on the results for the first few days. Since the new version shows up automatically in the frame in <a href="http://gameswithwords.fieldofscience.com/2014/05/the-english-grammars-of-world.html" target="_blank">the previous post</a>, I haven't added it in here. And you can get a better look at it <a href="http://gameswithwords.org/WhichEnglish/dialect_results.html" target="_blank">on the website</a>.<br />
<br />
The biggest difference is that also added several "dialects" for non-native speakers of English. That is, I added five new dialects, one each for people whose first language was Spanish, German, Portuguese, Dutch, or Finnish. I'll be adding more of these dialects in the future, but those just happen to be the groups for which we have a decent number of respondents.<br />
<br />
As you can see, the algorithm finds that American & Canadian speakers are more likely one another than they are like anyone else. Similarly, English, Irish, Scottish, and Australian speakers are more likely one another than anyone else. And the non-native English speakers also form a group. I'll leave you to explore the more fine-grained groupings on your own.<br />
<br />
If you are wondering why New Zealanders are off by themselves, that's mostly because we don't have very many of them, and the algorithm has difficulty classifying dialects for which there isn't much data. Same for Welsh English, South African English, and Black Vernacular English. So if you know people who speak any of those dialects...GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com5tag:blogger.com,1999:blog-7701757403364514168.post-1833654591438255842014-05-20T10:00:00.000-04:002014-05-26T17:35:27.787-04:00The English Grammars of the WorldIt's widely observed that not everybody speaks English the same way. Depending on where you grew up, you might say <a href="http://spark.rstudio.com/jkatz/Data/comp-50.png"><i>y'all</i>, <i>you guys</i>, or just <i>you</i></a>. You might pronounce <i>grocery </i><a href="http://spark.rstudio.com/jkatz/Data/comp-36.png">as if it were "groshery" or "grossery."</a> There have been some excellent, fine-grained studies of how these aspects of English vary across the United States and elsewhere, <a href="http://spark.rstudio.com/jkatz/DialectMap/">such as this one</a>.<br />
<br />
But vocabulary and pronunciation aren't the only things that vary across different dialects of English. We are in the midst of a soft launch of a <a href="http://gameswithwords.org/WhichEnglish/">new project</a> which will, among things, help map out the differences in English grammar around the world.<br />
<br />
I put together a visualization of early results below (you may want to load it in its <a href="http://gameswithwords.org/WhichEnglish/dialect_results.html" target="_blank">own page</a> -- depending on your browser, the embedded version below may not work). You can use this graphic to explore the similarities among nine English dialects (American, Canadian, English English, Irish, New Zealandish, Northern Irish, Scottish, and South African).<br />
<br />
As more results come in (about other dialects like Ebonics and Welsh, about specific parts of America or Canada, etc.), I'll be updating this graphic. So please take the survey and then check back in soon.<br />
<br />
<iframe src="http://gameswithwords.org/WhichEnglish/dialect_results.html" style="height: 650px; width: 650px;"></iframe><br />
<br />
Load the graphic directly <a href="http://www.gameswithwords.org/WhichEnglish/dialect_results.html">here</a>.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-35423752664333620592014-02-05T14:23:00.000-05:002014-02-05T14:23:24.188-05:00Doing your homework<div>
<span style="background-color: white; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 13px; line-height: 17.940000534057617px;">I just finished a radio interview about birth order.</span></div>
<span style="background-color: white; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 13px; line-height: 17.940000534057617px;"><div>
<span style="background-color: white; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 13px; line-height: 17.940000534057617px;"><br /></span></div>
Apparently not very much research goes into booking guests for radio & TV shows. Lately, I've been getting at least one interview request a month to talk about birth order. And every time they are disappointed that I can't tell them about how birth order affects personality, that there's little evidence to suggest it does. They *wouldn't* be surprised if they read *anything* that I had written or said on the topic. (Well, except for that FOX interview, which was edited to make it look like I said the exact opposite of what I actually said.)</span><div>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="background-color: white; line-height: 17px;"><br /></span></span></div>
<div>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="background-color: white; line-height: 17px;">It's been making me think I should do more birth order research, just so I have something to say at these interviews.</span></span><br /></div>
GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-39182703588871881362014-01-27T10:39:00.000-05:002014-01-27T10:39:00.606-05:00Calling all citizen scientistsSciStarter would like to know more about your experiences with Citizen Science. They are running a survey (<a href="https://www.surveymonkey.com/s/DW5YFTM">here</a>) in preparation for a workshop at the Citizen CyberScience Summit in London next month.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-801307664710093352014-01-09T10:43:00.000-05:002014-01-09T10:43:14.043-05:00More Citizens, More Science<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEsU_azm9EdmIraTwhuoLq8Idso9xSFZjGsggF5DYIK0MExCMaeu1c6DJUD7AqILVOPTln__Ut2VuuX51MtAKd-94onUeiB1vENib-bP7UZIgzt0VVXnjTNQ9DSS-1pKvDoukej6xAEzSu/s1600/Citizen.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEsU_azm9EdmIraTwhuoLq8Idso9xSFZjGsggF5DYIK0MExCMaeu1c6DJUD7AqILVOPTln__Ut2VuuX51MtAKd-94onUeiB1vENib-bP7UZIgzt0VVXnjTNQ9DSS-1pKvDoukej6xAEzSu/s1600/Citizen.jpg" height="320" width="167" /></a><br />
For the last couple years, most articles about <a href="http://en.wikipedia.org/wiki/Citizen_science">Citizen Science</a> -- in which amateurs contribute to scientific projects -- have been hagiography. These articles were nearly exclusively <i>Ra! Ra!</i>, all about the exciting new development.<br />
<br />
It seems that we've matured a bit as a field, because lately I've run across a couple articles that, while still being positive overall, have laid out some real criticism. For instance, in <a href="http://harvardmagazine.com/2014/01/popular-science">an article in <i>Harvard Magazine</i></a>, Katherine Xue concludes with the worry that citizen science may be less about involving the public and more about cheap labor (full disclosure: I was interviewed for and appear in this article). Many citizen science projects, she notes, are little more than games or, worse, rote labor, with little true engagement for the volunteer in the scientific mission.<br />
<br />
Similarly, in <a href="http://www.theguardian.com/science/political-science/2014/jan/06/placing-citizens-at-the-heart-of-citizen-science">a much-tweeted article</a> at <i>The Guardian</i>, Michelle Kilfoyle and Hayley Birch write, "Who really benefits the most from [citizen science]: the amateurs or the professionals? … Most well-known initiatives are the big crowdsourcing projects: big on the number of participants but not necessarily the level of participation."<br />
<br />
<b>Introducing the VerbCorner Forum</b><br />
<b><br /></b>
These articles resonated with me. Ever since we launched <a href="http://gameswithwords.org/VerbCorner/">VerbCorner</a>, our citizen science project looking at the structure of language, meaning, and thought, we've wanted to find additional ways to get our volunteers involved in the science and get more out of participation. VerbCorner is very much a crowdsourcing project -- most of what volunteers do on the site is contribute labor. We've always had this blog, where people could learn more about the project, but that's not especially interactive.<br />
<br />
To that end, we've added a <a href="http://gameswithwords.org/VerbCorner/forum.php">forum</a> where anyone and everyone involved in the project can discuss the project, offer suggestions, debate the science, and discuss anything related (syntax, semantics, etc.). We have high hopes for this forum. Over the years, I have gotten a lot of emails from participants in the various projects at GamesWithWords.org, emails with questions about the projects, ideas for new experiments, and -- all too often -- reports of bugs or type-os. These emails have been extremely useful, and in a few cases have even led to entirely new research directions. But email is a blunt instrument, and I expect that for everyone who has emailed, at least ten others had similar comments but never got around to tracking down our email address.<br />
<br />
I hope to see you on the forum!GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-56067634115431062012014-01-07T10:00:00.000-05:002014-01-07T10:00:03.198-05:00A Great Year for GamesWithWords.orgUnique visitors at GamesWithWords.org were up 76% in 2013 over the previous year. That's after several years of fairly steady traffic.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_AsskLsdo7ponJNfT0UFVhT77P5_OpzyA7t6MxdN6Z2tzmJXpkx8LNgSRpM7kibBSDOsJXRU1zIR26FV8eZb3zsk1dgKg1XCHguUnfaZ2ZXi3jRqa2o0tIITLiKfL_OQcgqvtyAtHk2jH/s1600/Traffic.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_AsskLsdo7ponJNfT0UFVhT77P5_OpzyA7t6MxdN6Z2tzmJXpkx8LNgSRpM7kibBSDOsJXRU1zIR26FV8eZb3zsk1dgKg1XCHguUnfaZ2ZXi3jRqa2o0tIITLiKfL_OQcgqvtyAtHk2jH/s1600/Traffic.png" height="219" width="640" /></a></div>
<br />
Meanwhile, <a href="http://www.gameswithwords.org/Hartshorne/papers/HartshorneSudoUruwashi_ICxLing.pdf">two</a> journal <a href="http://www.gameswithwords.org/Hartshorne/papers/Hartshorne_LCP_WhatIsIC.pdf">papers</a> and a conference <a href="http://www.gameswithwords.org/Hartshorne/posters/emnlp_2013_HartshorneBonialPalmer_final.pdf">paper</a> involving data collected at GamesWithWords.org were accepted (and two more are currently under review). Many thanks to everyone who participated and otherwise helped out!GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-78262806901401545292013-12-17T15:09:00.001-05:002014-01-27T10:10:45.481-05:00Results (Round 1): Crowdsourcing the Structure of Meaning & Thought<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVz5SJssEZ-ZW2-RB29Q-8aPZkJuGc5YK76d0B8t2iEodT4pALKHHvrXhaQhyphenhyphenGfHbt7dKsQFQni6RXumQen928pj5rT1rtQzW5h1RJbzQjCuJdsuiDhWy27sMgYv2oADCvJYhMMX41u4JC/s1600/Screen+Shot+2013-12-17+at+10.50.15+AM.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVz5SJssEZ-ZW2-RB29Q-8aPZkJuGc5YK76d0B8t2iEodT4pALKHHvrXhaQhyphenhyphenGfHbt7dKsQFQni6RXumQen928pj5rT1rtQzW5h1RJbzQjCuJdsuiDhWy27sMgYv2oADCvJYhMMX41u4JC/s400/Screen+Shot+2013-12-17+at+10.50.15+AM.png" height="306" width="400" /></a></div>
Language is a device for moving a thought from one person's head into another's. This means to have any real understanding of language, we also need to understand thought. This is what makes work on language exciting. It is also what makes it hard.<br />
<br />
With the help of over 1,500 <a href="http://en.wikipedia.org/wiki/Citizen_science">Citizen Scientists</a> working through our <a href="http://gameswithwords.org/VerbCorner/">VerbCorner</a> project, we have been making rapid progress.<br />
<br />
<b>Grammar, Meaning, & Thought</b><br />
<b><br /></b>
You can say <i>Albert hit the vase</i> and <i>Albert hit at the vase</i>. You can say <i>Albert broke the vase </i>but you <b>can't</b> say <i>Albert broke at the vase</i>. You can say <i>Albert sent a book to the boarder</i> [a person staying at a guest house] or <i>Albert sent a book to the border </i>[the line between two countries], but while you can say <i>Albert sent the boarder a book</i>, you can't say <i>Albert sent the border a book</i>. And while you say <i>Albert frightened Beatrice</i> -- where Beatrice, the person experiencing the emotion, is the object of the verb -- you must say <i>Beatrice feared Albert</i> -- where Beatrice, the person experiencing the emotion, is now the subject.<br />
<i><br /></i>
<a href="http://hdwallpaperspictures.com/wp-content/uploads/2013/11/Doggy-dogs-27377981-1595-15961.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://hdwallpaperspictures.com/wp-content/uploads/2013/11/Doggy-dogs-27377981-1595-15961.jpg" height="200" width="199" /></a>How do you know which verb gets used which way? One possibility is that it is random, and this is just one of those things you must learn about your language, just like you have to learn that the animal in the picture on the left is called a "dog" and not a "<a href="http://www.spanishdict.com/translate/perro">perro</a>", "<a href="http://dictionary.reverso.net/english-chinese/dog">xiaogou</a>," or "<a href="http://dictionary.reverso.net/russian-english/%D1%81%D0%BE%D0%B1%D0%B0%D0%BA%D0%B0">sobaka</a>." This might explain why it's hard to learn language -- so hard that non-human animals and machines can't do it. In fact, it results in a learning problem so difficult that many researchers believe it would be impossible, even for humans (see especially work on <a href="http://books.google.com/books?id=me11YSch_xcC&printsec=frontcover#v=onepage&q&f=false">Baker's Paradox</a>).<br />
<i><br /></i>
<a href="http://books.google.com/books?id=6wIZWOrcBf8C&printsec=frontcover&dq=Beth+Levin+verb+alternations&hl=en&sa=X&ei=8JawUrSVBYmzsQSuqoGADA&ved=0CC8Q6AEwAA#v=onepage&q=Beth%20Levin%20verb%20alternations&f=false">Many</a> <a href="http://books.google.com/books?id=HDUvYcW5n6QC&printsec=frontcover&dq=Pinker+Learnability+and+cognition&hl=en&sa=X&ei=B5ewUqeTCavIsASHjYHoDQ&ved=0CC0Q6AEwAA#v=onepage&q=Pinker%20Learnability%20and%20cognition&f=false">researchers</a> <a href="http://books.google.com/books?id=7wbYlHis6OEC&printsec=frontcover&dq=Jackendoff+Semantic+Structures&hl=en&sa=X&ei=35awUqyjEInQsASEp4HoCw&ved=0CC8Q6AEwAA#v=onepage&q=Jackendoff%20Semantic%20Structures&f=false">have</a> suspected that there are patterns in terms of which verbs can get used in which ways, explaining the structure of language and how language learning is possible, as well as shedding light on the structure of thought itself. For instance, the difference (it is argued) between <i>Albert hit the vase</i> and <i>Albert hit at the vase</i> is that the latter sentence means that Albert hit the vase ineffectively. You can't say <i>Albert broke at the vase</i> because you can't ineffectively break something: It is either broken or not. The reason you can't say <i>Albert sent the border a book</i> is that this construction means that the border owns the book, which a border can't do -- borders aren't people and can't own anything -- but a boarder can. The difference between <i>Albert frightened Beatrice</i> and <i>Beatrice feared Albert</i> is that the former describes an event that happened in a particular time and place (compare <i>Albert frightened Beatrice yesterday in the kitchen</i> with <i>Beatrice feared Albert yesterday in the kitchen</i>).<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi559-oaVrqVEFI2SVJ28f6mesJGjY4of0cZgGmd9PRsXF0HxhuJYjCTq1KHgnqtqmWQz19AyEHZa8L-oX9wa157FTW22m3Vs50kexsONNSBlop2b8O4io7cGRRvmBe2V2gnAypHni0UlfM/s1600/stuff.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi559-oaVrqVEFI2SVJ28f6mesJGjY4of0cZgGmd9PRsXF0HxhuJYjCTq1KHgnqtqmWQz19AyEHZa8L-oX9wa157FTW22m3Vs50kexsONNSBlop2b8O4io7cGRRvmBe2V2gnAypHni0UlfM/s1600/stuff.jpg" height="200" width="130" /></a><br />
<br />
When researchers look at the aspects of meaning that matter for grammar across different languages, many of the same aspects pop up over and over again. Does the verb describe something changing (<i>break </i>vs. <i>hit</i>)? Does it describe something only people can do (<i>own, know, believe </i>vs. <i>exist</i>, <i>break</i>, <i>roll</i>)? Does it describe<i> </i>an event or a state (<i>frighten</i> vs. <i>fear</i>)? This is too suspicious of a pattern to be accidental. Researchers like <a href="http://stevenpinker.com/">Steven Pinker</a> have argued that language cares about these aspects of meaning because these are basic distinctions our brain makes when we think and reason about the world (see <a href="http://books.google.com/books?id=jylSITT9ZNUC&printsec=frontcover&dq=Pinker+Stuff+of+Thought&hl=en&sa=X&ei=IZuwUp3xOIK3sASsx4CgBw&ved=0CDwQ6AEwAA#v=onepage&q=Pinker%20Stuff%20of%20Thought&f=false" style="font-style: italic;">Stuff of Thought</a>). Thus, the structure of language gives us insight into the structure of thought.<br />
<br />
<b>The Question</b><br />
<br />
The theory is very compelling and is exciting if true, but there are good reasons to be skeptical. The biggest one is that there simply isn't that much evidence one way or another. Although a few grammatical constructions have been studied in detail (in recent years, this work has been spearheaded by <a href="http://pcwww.liv.ac.uk/~ambridge/">Ben Ambridge</a> of the University of Liverpool), the vast majority have not been systematically studied, even in English. Although evidence so far suggests that which verbs go in which grammatical constructions is driven primarily or entirely by meaning, skeptics have argued that is because researchers so far have focused on exactly those parts of language that are systematic, and that if we looked at the whole picture, we would see that things are not so neat and tidy.<br />
<br />
The problem is that no single researcher -- nor even an entire laboratory -- can possibly investigate the whole picture. Checking every verb in every grammatical construction (e.g., <i>noun verb noun </i>vs. <i>noun verb <b>at</b> noun</i>, etc.) for every aspect of meaning would take one person the rest of her life.<br />
<br />
<b>CrowdSourcing the Answer</b><br />
<br />
<a href="http://blog.hbs.edu/hbsinov8/wp-content/uploads/2012/02/CrowdSourcing-Logo1.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://blog.hbs.edu/hbsinov8/wp-content/uploads/2012/02/CrowdSourcing-Logo1.jpg" height="188" width="320" /></a>Last May, <a href="http://gameswithwords.org/VerbCorner/">VerbCorner</a> was launched to solve this problem. For the first round of the project, we posted questions about 641 verbs and six different aspects of meaning. By October 18th, 1,513 volunteers had provided 117,584 judgments, which works out to 3-4 people per sentence per aspect of meaning. That was enough data to start analyzing.<br />
<br />
As predicted, there is a great deal of systematicity in the relationship between meaning and grammar (for details on the analysis, see the next section). These results suggest that the relationship between grammar and meaning may indeed be very systematic, helping to explain how language is learnable at all. It also gives us some confidence in the broad project of using language as a window into how the brain thinks and reasons about the world. This is important, because the mind is not easy to study, and if we can leverage what we know about language, we will have learned a great deal. As we test more verbs and more aspects of meaning -- I recently added an additional aspect of meaning and several hundred new verbs -- that window will be come clearer and clearer.<br />
<br />
Unless, of course, it turns out that not all of language is so systematic. While our data so far represent a significant proportion of all research to date, it's only a tiny fraction of English. That is what makes research on language so hard: there is so much of it, and it is incredibly complex. But with the support of our volunteer Citizen Scientists, I am confident that we will be able to finish the project and launch a new phase of the study of language.<br />
<br />
That brings up one additional aspect of the results: It shows that this project is possible. Citizen Science is <a href="http://blogs.scientificamerican.com/mind-guest-blog/2013/09/23/citizen-scientists-decode-meaning-memory-and-laughter/">rare in the study of the mind</a>, and many of my colleagues doubted that amateurs could provide reliable results. In fact, by the standard measures of reliability, the information our volunteers contributed is very reliable.<br />
<br />
Of course, checking for a systematic relationship between grammar and meaning is only the first step. We'd also liked to understanding which verbs and grammatical constructions have which aspects of meaning and why, and leverage this knowledge into understanding more about the nature of thought. Right now, we still don't have enough data to have exciting new conclusions (for exciting old conclusions, see Pinker's <i>Stuff of Thought</i>). I expect I'll have more to say about that after we complete the next phase of data collection.<br />
<br />
<b>Details of the Analysis</b><br />
<br />
Here is how we did the analyses. If meaning determines which grammatical constructions a given verb can appear in, then you would expect that all the verbs that appear in the same set of frames should be the same in terms of the core aspects of meaning discussed above. So if one of those verbs describes, for instance, physical contact, then all of them should.<br />
<br />
Helpfully, the <a href="http://verbs.colorado.edu/verb-index/index.php">VerbNet project</a> -- which was built on <a href="http://books.google.com/books?id=6wIZWOrcBf8C&printsec=frontcover&dq=Beth+Levin+alternations&hl=en&sa=X&ei=qqewUr7YDbLNsQTwjYDoDw&ved=0CDEQ6AEwAA#v=onepage&q=Beth%20Levin%20alternations&f=false">earlier work</a> by Beth Levin -- has already classified over 6,000 English verbs according to which grammatical constructions they can appear in. The 641 verbs posted in the first round of the VerbCorner project consisted of all the verbs from 11 of these classes.<br />
<br />
So is it the case that in a given class, all the verbs describe physical contact or all of them do not? One additional complication is that, as I described above, the grammatical construction itself can change the meaning. So what I did was count what percentage of verbs from the same class have the same value for a given aspect of meaning for each grammatical construction, and then I averaged over those constructions.<br />
<br />
The "Explode on Contact" task in VerbCorner asked people to determine whether a given sentence (e.g., <i>Albert hugged Beatrice</i>) described contact between different people or things. Were the results for a given verb class and a given grammatical construction? Several volunteers checked each sentence. If there was disagreement among the volunteers, I used whatever answer the majority had chosen.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9n7P6-wPP3Pnt3HN0bd2W4w3O5j-_7bYAnPIs9euHOr-N9VnyVVklu0w6rx6EQBUEoKMauxPCkoWDBUfefrfj9d0PN-qIhFCI5ab9NDgdxvm9ebmJv290bWUmLH6sIMHJSobclNn2A58l/s1600/Contact+copy.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9n7P6-wPP3Pnt3HN0bd2W4w3O5j-_7bYAnPIs9euHOr-N9VnyVVklu0w6rx6EQBUEoKMauxPCkoWDBUfefrfj9d0PN-qIhFCI5ab9NDgdxvm9ebmJv290bWUmLH6sIMHJSobclNn2A58l/s400/Contact+copy.jpg" height="297" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
This graph shows the degree of consistency by verb class (the classes are numbered according to their VerbNet number), with 100% being maximum consistency. You can see that all eleven classes are very close to 100%. Obviously, exactly 100% would be more impressive, but that's extremely rare to see when working with human judgments, simply because people make mistakes. We addressed this in part by having several people check each sentence, but there are so many sentences (around 5,000), that simply by bad luck sometimes several people will all make a mistake on the same sentence. So this graph looks as close to 100% as one could reasonably expect. As we get more data, it should get clearer.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Results were similar for other tasks. Another one looked at whether the sentence described someone applying force (pushing, shoving, etc.) to something or someone else:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2KPOWLxbQ33S5_KNoP3i1Mmy77aXraIssg8YVWwGbR9tLRfRNxM-cVae7rwt0cEH7cviiXWij-unKfP9cCfKocAGqbmHHn1c5Fk5l4f_8BO9w_5IKxXPgtG28-bva2BV6NWBvuBsgE36d/s1600/ApplicationOfForce.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2KPOWLxbQ33S5_KNoP3i1Mmy77aXraIssg8YVWwGbR9tLRfRNxM-cVae7rwt0cEH7cviiXWij-unKfP9cCfKocAGqbmHHn1c5Fk5l4f_8BO9w_5IKxXPgtG28-bva2BV6NWBvuBsgE36d/s400/ApplicationOfForce.jpg" height="308" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Maybe everything just looks very consistent? We actually had a check for that. One of the tasks measures whether the sentence describes something that is good, bad, or neither. These is no evidence that this aspect of meaning matters for grammar (again, the hypothesis is not that every aspect of meaning matters -- only certain ones that are particularly important for structuring thought are expected to matter). And, indeed, we see much less consistency:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9BdULo3uqcYvvjgmB4OuW8Yldv8XsiXfkQc_BaKa_kHEyt7bUap21Edc0Tc6iD9lfAX2_j6F_2pRcBb7Zry42TLhOElWdLDittE3u7M6rgcvsYiI6nOAqjwL6Pq1EjVaUgYK0zu1D5O0j/s1600/Evaluation.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9BdULo3uqcYvvjgmB4OuW8Yldv8XsiXfkQc_BaKa_kHEyt7bUap21Edc0Tc6iD9lfAX2_j6F_2pRcBb7Zry42TLhOElWdLDittE3u7M6rgcvsYiI6nOAqjwL6Pq1EjVaUgYK0zu1D5O0j/s400/Evaluation.jpg" height="297" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Notice that there is still some consistency, however. This seems to be mostly because most sentences describe something that is neither good nor bad, so there is a fair amount of essentially accidental consistency within each verb class. Nonetheless, this is far less consistency that what we saw for the other five aspects of meaning studied.</div>
GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com3tag:blogger.com,1999:blog-7701757403364514168.post-53058936868682346272013-12-17T10:46:00.000-05:002013-12-17T10:47:18.577-05:00Citizen Science in Harvard MagazineA nice, extended article on recent projects, covering a wide range -- including GamesWithWords.org. <a href="http://harvardmagazine.com/2014/01/popular-science">Check it out</a>.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-81915323853100956302013-10-04T10:08:00.002-04:002013-10-04T12:12:45.026-04:00Science Mag studies science. Forgets to include control group.Today's issue of <i>Science </i>carries the <a href="http://www.sciencemag.org/content/342/6154/60.full">most meta sting operation</a> I have ever seen. John Bohannon reports a study of open access journals, showing lax peer review standards. He sent 304 fake articles with obvious flaws to 304 open access journals, more than half of which were accepted.<br />
<br />
The article is written as a stinging rebuke of open access journals. Here's the interesting thing: There's no comparison to traditional journals. For all we know, open access journals actually have *stricter* peer review standards than traditional journals. We all suspect not, but suspicion isn't supposed to count as evidence in science. Or in <i>Science</i>.<br />
<br />
So this is where it gets meta: <i>Science</i> -- which is not open access -- published an obviously flawed article about open access journals publishing obviously flawed articles.<br />
<br />
It would be even better if Bohannon's article had run in the "science" section of <i>Science</i>, rather than in the news section, where it actually ran, but hopefully we can agree that <i>Science</i> can't absolve itself of checking its articles for factualness and logical coherence just by labeling them "news".GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-46525989224438851002013-09-25T11:00:00.000-04:002013-09-25T11:00:04.623-04:00Titling<br />
I have never been good at coming up with titles for articles. When writing for newspapers or magazines, I usually leave it up to the editor. There is some danger that comes with this, however.<div>
<br /></div>
<div>
Last week, I <a href="http://www.scientificamerican.com/article.cfm?id=how-to-understand-the-deep-structures-of-language">wrote a piece</a> for Scientific American about similarities across languages. This piece was <a href="http://www.salon.com/2013/09/21/how_to_understand_the_deep_structures_of_language_partner/">then picked up by Salon</a>, which re-ran the article under a new title:</div>
<blockquote class="tr_bq">
Chomsky's "Universal Language" is incomplete. Chomsky's theory does not adequately explain why different languages are so similar.</blockquote>
I agree that this is snappier than any title I would have come up with. It's also perhaps a bit snappier than the one Scientific American used. It's also dead wrong. For one, there is no such thing as Chomsky's "Universal Language." Or if there is, presumably it is love. Or maybe mathematics. Or maybe music. The term is "Universal Grammar."<br />
<br />
If you squint, the subtitle isn't exactly wrong. In the article, I do claim that standard Universal Grammar theory's explanation of similarities across languages isn't quite right. But the title implies that UG suggests that languages are not that similar, whereas the real problem with UG is that -- at least on standard interpretations -- it suggests that languages should be more similar than they actually are.<br />
<br />
I sent in a letter to "corrections" at Salon, and the title has now been switched to something more correct. The moral of the story? Apparently writing good titles really is just very hard.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-8544916852660324092013-09-24T10:00:00.000-04:002013-09-24T13:03:12.851-04:00GamesWithWords on Scientific AmericanOver the last week, ScientificAmerican.com has published two articles by me. The most recent, "<a href="http://blogs.scientificamerican.com/mind-guest-blog/2013/09/23/citizen-scientists-decode-meaning-memory-and-laughter/" target="_blank">Citizen Scientists decode meaning, memory and laughter</a>," discusses how citizen science projects -- science projects involving collaborations between professional scientists and amateur volunteers -- are now being used to answer questions about the human mind.<br />
<br />
<blockquote class="tr_bq">
Citizen Science – projects which involve collaboration between professional scientists and teams of enthusiastic amateurs — is big these days. It’s been great for layfolk interested in science, who can now not just read about science but participate in it. It has been great for scientists, with numerous mega-successes like Zooniverse and Foldit. Citizen Science has also been a boon for science writing, since readers can literally engage with the story.</blockquote>
<blockquote class="tr_bq">
However, the Citizen Science bonanza has not contributed to all scientific disciplines equally, with many projects in zoology and astronomy but less in physics and the science of the mind. It is maybe no surprise that there have been few Citizen Science projects in particle physics (not many people have accelerators in their back yards!), but the fact that there has been very little Citizen Science of the mind is perhaps more remarkable.</blockquote>
<br />
The article goes on to discuss three new mind-related citizen science projects, including our own VerbCorner project.<br />
<br />
The second, "<a href="http://www.scientificamerican.com/article.cfm?id=how-to-understand-the-deep-structures-of-language" target="_blank">How to understand the deep structures of language</a>," describes some really exciting work on how to explain linguistic universals -- work that was conducted by colleagues of mine at MIT.<br />
<blockquote class="tr_bq">
In an exciting recent paper, Ted Gibson and colleagues provide evidence for a design-constraint explanation of a well-known bias involving case endings and word order. Case-markers are special affixes stuck onto nouns that specify whether the noun is the subject or object (etc.) of the verb. In English, you can see this on pronouns (compare "she talked with her"), but otherwise, English, like most SVO languages (languages where the typical word order is Subject, Verb, Object) does not mark case. In contrast, Japanese, like most SOV languages (languages where the typical word order is Subject, Object, Verb) does mark case, with -wa added to subjects and -o added to direct objects. "Yasu saw the bird" is translated as "Yasu-wa tori-o mita" and "The bird saw Yasu" is translated as "Tori-wa Yasu-o mita." The question is why there is this relationship between case-marking and SOV word order.</blockquote>
The article ran in the Mind Matters column, which invites scientists to write about the paper that came out in the last year that they are most excited about. It was very easy for me to choose this one.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com2tag:blogger.com,1999:blog-7701757403364514168.post-64764245171439619292013-09-12T14:39:00.001-04:002013-09-12T14:39:34.818-04:00Language and Memory ReduxOne week only: If you did not do our <a href="http://gameswithwords.org/LanguageAndMemory/" target="_blank">Language and Memory</a> task when it was running earlier this year, now is your chance. We just re-launched it to collect some additional data.<br />
<br />
I expect we'll have enough data without a week to finish this line of studies, rewrite the paper (this is a follow-up experiment that was requested by peer reviewers), and also post the full results here.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com0tag:blogger.com,1999:blog-7701757403364514168.post-91573329196987211072013-09-10T09:00:00.000-04:002013-09-12T20:09:51.856-04:00Вы понимаете по-русски?У нас <a href="http://gameswithwords.org/Diva/" target="_blank">новый русский эксперимент</a>. Большинство психолингвистов занимаются английским. Мы хотим узнать больше об остальних. Не волнуйтесь -- я не сам перевёл эксперимент. Перевела его настоящая рускоязычная!<br />
<br />
If you didn't understand that, that's fine. We're recruiting participants for a new experiment in Russian. Apparently you aren't eligible. :)<br />
<br />
Much of the research on language is done on a single language: English. In part, that's because many researchers happen to live in English-speaking countries. The great thing about the Internet is we are freed from the tyranny of geography.GamesWithWordshttp://www.blogger.com/profile/15107067137612954306noreply@blogger.com3