Around the Internet - 8/31

Publication
A warning about the perils of preprint repositories.

Statistical evidence that writing book chapters isn't worth the effort. (Though caveat: the author also doesn't find evidence of higher citation rates for review papers in journals, which I had thought was well-established.)

One person who finds things to like in the publication process (I know, I don't link to these often).

Neuroskeptic argues that we don't necessarily want to increase replication, just replicability. (Agreed, but how do we know if replicability rates are high enough without conducting replications?)

Language
Did Chris Christie really talk about himself too much in Tampa? 

Other Cognitive Science
Cognitive load disrupts implicit theory of mind processing. So maybe the reason young children succeed at implicit tasks isn't because those tasks don't require executive processing (whether they require less is still up for grabs).

Lying with statistics

One of the most concise explanations of why your units of measurement matter, courtesy of XKCD:

Revision, Revision, Revision

I have finally been going through the papers in the Frontiers Special Topic on publication and peer review in which my paper on replication came out. One of the arguments that appears in many of these papers (like this one)* -- and many discussions of the review process, is that when papers are published, they should be published along with the reviews.

My experience with the process -- which I admit is limited -- is that you submit a paper, reviewers raise concerns, and you only get published if you can revise the manuscript so as to address those concerns (which may include new analyses or even new experiments). At that stage, the reviews are a historical document, commenting on a paper that no longer exists. This may be useful to historians of science, but I don't understand how it helps the scientific process (other than, I suppose, transparency is a good thing).

So these proposals only make sense to me if it is assumed that papers are *not* typically revised in any meaningful way based on review. That is, reviews are more like book reviews: comments on a finished product. Of my own published work, three papers were accepted more-or-less as is (and frankly I think the papers would have benefited from more substantial feedback from the reviewers). So there, the reviews are at least referring to a manuscript very similar to the one that appeared in print (though they did ask me to clarify a few things in the text, which I did).

Other papers went through more substantial revision. One remained pretty similar in content, though we added a whole slew of confirmatory analyses that were requested by reviewers. The most recent paper actually changed substantially, and in many ways is a different -- and much better! -- paper than what we originally submitted. Of the three papers currently under review, two of them have new experiments based on reviewer comments, and the other one has an entirely new introduction and general discussion (the reviewers convinced me to re-think what I thought the paper was about). So the reviews would help you figure out which aspects of the paper we (the authors) thought of on our own and which are based on reviewer comments, but even then that's not quite right, since I usually get comments from a number of colleagues before I make the first submission. There are of course reviews from the second round, but that's often just from one or two of the original reviewers, and mostly focuses on whether we addressed their original concerns or not.

So that's my experience, but perhaps my experience is unusual. I've posted a poll (look top right). Let me know what your experience is. Since this may vary by field, feel free to include comments to this post, saying what field you are in.

---
*To be fair, this author is describing a process that has actually been implemented for a couple Economics journals, so apparently it works to (at least some) people's satisfaction.

Have you seen me before?

I have been using PCA to correct blink artifact in an EEG study that I am presenting at AMLaP in a couple weeks. Generally, I think I've gotten pretty good at detecting blinks. I do see other things that look like artifact but which I don't understand as well. For instance, look at this channel plot:

(You should be able to increase the size of the picture by opening it in a new window). So this looks a bit like a blink, but it's in the wrong place entirely. This is a 128 electrode EGI cap, with the electrodes listed sequentially (the top one is electrode 1 and the bottom is electrode 124 -- I don't use electrodes 125-128 because they tend not to have good contact with the skin).

The way EGI is laid out, the low-numbered electrodes and high-numbered electrodes are in the front, whereas the middle-numbered electrodes are in the back (check this picture), So basically what I'm seeing is being generated in the back of the head. Actually, the back left of the head, according to my PCA:

In this, the top left panel shows the localization of the signal. The top right panel shows which trials the signal occurred in. The power spectrum (bottom panel) is also quite odd. I'm going ahead and removing this component, because it's clearly artifact (the amplitude looks way too large to be true EEG), and it affects so many trials that I can't just exclude them without excluding the participant. But I'd really like to know what this is. Because maybe I *should* be excluding the participant.

So...has anyone seen something like this before?

For those wondering...

Using PCA, I was able to get rid of this artifact fairly cleanly. Here's is an imagine before removal, with the 124 electrodes stacked on one another:

You can see that strange artifact -- which looks like a blink but not quite as smooth as your typical blink -- very easily in these four trials.

Here are the same four trials after I subtracted that component, plus another component that probably is blink-related (there were two, classic-looking blinks in my data; the component above found both of those *and* those two blinks; the other component found only the two classic blinks):

You can see that the odd artifact is gone from all four trials, both otherwise, things look very similar.

Around the Internet - 7/30/2012

Citations

There have been a bunch of posts lately on citations and the Impact Factor. I started with these two posts by DrugMonkey. These posts have links to others in the chain, which you can follow. Here's a slightly older post (from late July) on reasons to self-cite.

Next topic

So I didn't actually see anything else interesting this week. Possibly because I've been trying to streamline a bootstrapping analysis (which I may blog about when I finally get it done). Early in the process, I tried to estimate how long it would take for the script to run and realized it was about 1 week for each analysis, of which I have several to do. So I started hurriedly looking for ways to speed it up...

New source of post-doctoral funding

NSF has just announced what appears to be a new post-doctoral fellowship. The document linked to lists two different tracks: Broadening Participation and Interdisciplinary Research in Behavioral and Social Sciences. It is the second one that seems to be new. Here's the heart of the description:

Track 2. Interdisciplinary Research in Behavioral and Social Sciences (SPRF-IBSS): The SPRF-IBSS track aims to support interdisciplinary training where at least one of the disciplinary components is an SBE science ... The proposal must be motivated by a compelling research question (within the fields of social, behavioral and economic sciences) that requires an interdisciplinary approach for successful investigation. As a result, applicants should demonstrate the need for new or additional skills and expertise beyond his or her core doctoral experience to achieve advances in the proposed research. To acquire the requisite skills and competencies (which may or may not be within SBE sciences), a mentor in the designated field must be selected so that the postdoctoral research fellow and his or her mentor will complement, not reinforce, each other's expertise. 

What I get from this is that the fellowship will be particularly useful for someone with training in one field who wants to get cross-trained in another. Thinking close to home, this might be a psycholinguist who wants training in linguistics or computer science. This makes me think of the legendary IGERT program at UPenn, which trained a string of linguists to use psychological research methods, many of whom are now among my favorite researchers. Which is to say that this cross-training can be very productive.

Another bunch of retractions

It appears that a series of papers, written by a German business professor, are being retracted. This particular scandal doesn't seem to involve data fabrication, though. Instead, he is accused of double-publishing (publishing the same work in multiple journals) and also of making errors in his analyses (this lengthy article -- already linked to above -- discusses the issues in detail).

It's possible that I was not paying attention before, but there seems to be more publication scandals lately than I remember. When working on my paper about replication, I actually had to look pretty hard to find examples of retracted papers in psychology. That wouldn't be so difficult at the moment, after Hauser, Smeeters and Sanna.

If there is an increase, it's hopefully due not to an increase in fraud but an increase in vigilance, given the attention the issue has been getting lately.

Making up your data

Having finished reading the Simonsohn paper on detecting fraud, I have come to two conclusions:

1. Making up high-quality data is really hard. Part of the problem with making up data is that you have to introduce some randomness into it. If your study involves asking people how much they are willing to pay for a black t-shirt, you can't just write down that they all were willing to pay the average (say $12). You have to write down some variation ($12, $14, $7, $9, etc.).

The problem is that humans are notoriously bad at generating random number sequences. Simonsohn discusses this in terms of Tversky and Kahneman's famous, tongue-in-cheek paper "Belief in the law of small numbers." People think that random sequences should look roughly "average", even if the sample is small: Flip a coin 4 times, you should get 2 heads and 2 tails, when in fact getting 4 heads isn't all that improbable.

So your best bet, if you are making up data, is to use a computer program to generate it from your favorite distribution (the normal distribution would be a good choice in most cases). The problem is that data can have funny idiosyncrasies. One of the problems with the string of numbers I suggested above ($12, $14, $7, $9, etc.) is that humans like round numbers. So when people say what they are willing to pay for a t-shirt, what you should see is a lot of $10s, $20s and maybe some $5s and $15s. The numbers in my list are relatively unlikely.

The paper goes on to describe other problems as well. What I get from this is that making up data in a way that is undetectable is a lot of work, and you might as well actually run the study. So even leaving aside other reasons you might want to not commit fraud (ethics, desire for / belief in importance of knowledge, etc.), it seems sheer laziness alone should steer you the other direction.

2. The Dark Knight Rises is awesome. Seriously. Technically there was nothing about that in the paper, but I was thinking about the movie while reading the paper. Since I saw the show this morning, it's been hard to think of anything else. The most negative thing I can say about it is that it wasn't better than the last one, which is grading on a pretty steep curve.

Detecting fraud

Uri Simonsohn has now posted a working paper describing how he detected those two recent cases of data fraud. Should my other writing projects progress fast enough, I'll write about it soon. I'll also post links to any interesting discussions I come across.

A visual depiction of vision

Filed here, so I can use it next time I teach intro psychology:

What did we do before XKCD?

Update on Dragon Dictate

I recently bought a new computer, and Dragon Dictate is working much better on it, if not perfectly. And this is despite the fact that I have trained the new copy much less than the old one. One annoying/funny problem that keeps coming up: Dictate always transcribes "resubmission" as "recent mission". So, "Here's the news from the resubmission" becomes "Here's the news from the recent mission."

Since one can't be snarky in a response to a review...

I'll do it here. I am currently revising a paper for resubmission. On the whole, the reviews are fairly reasonable, with the exception of one cranky comment from a reviewer who complains that our literature review is woefully incomplete. This incompleteness seems to be our failure to cite one particular study. The reviewer writes

It is possible that this work is flawed, but it really should be discussed.

It does seem to be a relevant study and we would have cited it, had we known about it. Why didn't we know about it? Because it has never been published. It hasn't even been presented at any of the normal psycholinguistics conferences (though it has appeared at some linguistics conferences). Short of emailing every researcher who might be conducting a study that might be relevant, I'm not sure what this reviewer was expecting of us.

I'd also love to know what the folks who are obsessed with only citing studies published in peer-reviewed journals would say. (It's possible that some of these conferences it has been presented at have pretty thorough review procedures; I wouldn't know.)

The Psychologist on Replication

The Psychologist solicited opinions on the importance of replication from a number of researchers, including yours truly. See a preview here.

Eadweard J. Muybridge & Google Doodle

Today's Google Doodle is a fantastic tribute to Muybridge. I haven't found a permalink, but people looking after today can find it archived in a fashion on youtube.

Point-light walkers

By far the best point-light walker demonstration I've seen is at biomotiolab.ca. I'm classifying this as an illusion (see post label) because, of course, point-light walkers aren't really walking people -- they are just a few white dots moving around the screen. Comparing the male and female versions is particularly fun if you've ever wondered what exactly it is that makes for a stereotypical male or female stride.

It also appears that there is an experiment you can participate in if you want to help with this kind of research.