Is there a ‘basement’ for quirky psychological research?

Beware the Basement!
Beware the Basement!

 One thing I will never forget from my undergraduate training in psychology was the first lecture of my personality theory class. The professor started the lecture by informing us that he was quite sure that of the 200+ students in the lecture hall, the majority of us were probably majoring in psychology because we thought it would be neat to study sex, consciousness, psychedelics, paranormal experience, meditation, or the ilk. He then informed us this was a trap that befell almost all new psychology students, as we were all drawn to the study of the mind by the same siren call of the weird and wonderful human psyche. However he warned, we should be very, very careful not to reveal these suppressed interests until we were well established (I’m assuming he meant tenured) researchers- otherwise we’d risk being thrown into the infamous ‘basement of psychology’, never to be heard from again.

This colorful lecture really stuck with me through the years; I still jokingly refer to the basement whenever a more quirky research topic comes up. Of course I did a pretty poor job of following this advice, seeing as my first project as a PhD student involved meditation, but nonetheless I have repressed an academic interest in more risque topics throughout my career. And i’m not really actively avoiding them for fear of being placed in the basement – i’m more just following my own pragmatic research interests, and waiting for some day when I have more time and freedom to follow ideas that don’t directly tie into the core research line I’m developing.

But still. That basement. Does it really exist? In a world where papers about having full bladders renders us more politically conservative can make it into prestigious journals, or where scientists scan people having sex inside a scanner just to see what happens, or where psychologists seriously debate the possibility of precognition – can anything really be taboo? Or can we still distinguish from these flightier topics a more serious avenue of research? And what should be said about those who choose such topics?

Personally I think the idea of a ‘basement’ is largely a hold-over from the heyday of behaviorism, when psychologists were seriously concerned about positioning psychology as a hard science. Cognitivism has given rise to an endless bevy of serious topics that would have once been taboo; consciousness, embodiment, and emotion to name a few. Still, in the always-snarky twittersphere, one can’t but help feel that there is still a certain amount of nose thumbing at certain topics.

I think really, in the end, it’s not the topic so much as the method. Chris Frith once told me something to the tune of ‘in [cognitive neuroscience] all the truly interesting phenomenon are beyond proper study’. We know the limitations of brain scans and reaction times, and so tend to cringe a bit when someone tries to trot out the latest silly-super-human special interest infotainment paper.

What do you think? Is there a ‘basement’ for silly research? And if so, what defines what sorts of topics should inhabit its dank confines?

PubPeer – A universal comment and review layer for scholarly papers?

Lately I’ve had a plethora of discussions with colleagues concerning the possible benefits of a reddit-like “democratic review layer”, which would index all scholarly papers and let authenticated users post reviews subject to karma. We’ve navel-gazed about various implementations ranging from a full out reddit clone, a wiki, or even a full blown torrent tracker with rated comments and mass piracy. So you can imagine I was pleasantly surprised to see someone actually went ahead and put together a simple app to do exactly that.

Image

Pubpeer states that it’s mission is to “create an online community that uses the publication of scientific results as an opening for fruitful discussion.” Users create accounts using an academic email address and must have at least one first-author publication to join. Once registered any user can leave anonymous comments on any article, which are themselves subject to up/down votes and replies.

My first action was of course to search for my own name:

Image

Hmm, no comments. Let’s fix that:

Image

Hah! Peer review is easy! Just kidding, I deleted this comment after testing to see if it was possible. Ostensibly this is so authors can reply to comments, but it does raise some concerns that one can just leave whatever ratings you like on your own papers. In theory with enough users, good comments will be quickly distinguished from bad, regardless of who makes them.  In theory… 

This is what an article looks like in PubPeer with a few comments:

Image

Pretty simple- any paper can be found in the database and users then leave comments associated with those papers. On the one hand I really like the simplicity and usability of PubPeer. I think any endeavor along these lines must very much follow the twitter design mentality of doing one (and only one) thing really well. I also like the use of threaded comments and upvote/downvotes but I would like to see child comments being subject to votes. I’m not sure if I favor the anonymous approach the developers went for- but I can see costs and benefits to both public and anonymous comments, so I don’t have any real suggestions there.

What I found really interesting was just to see this idea in practice. While I’ve discussed it endlessly, a few previously unforeseen worries leaped out right away. After browsing a few articles it seems (somewhat unsurprisingly) that most of the comments are pretty negative and nit-picky. Considering that most early adopters of such a system are likely to be graduate students, this isn’t too surprising. For one thing there is no such entity as a perfect paper, and graduate students are often fans of these kind of boilerplate nit-picks that form the ticks and fleas of any paper. If comments add mostly doubt and negativity to papers, it seems like the whole commenting process would become a lot of extra work for little author pay-off, since no matter what your article is going to end up looking bad.

In a traditional review, a paper’s flaws and merits are assessed privately and then the final (if accepted) paper is generally put forth as a polished piece of research that stands on it’s on merits. If a system like PubPeer were popular, becoming highly commented would almost certainly mean having tons of nitpicky and highly negative comments associated to that manuscript. This could manipulate reader perceptions- highly commented PubPeer articles would receive fewer citations regardless of their actual quality.

So that bit seems very counter-productive to me and I am not sure of the solution. It might be something similar to establishing light top-down comment moderation and a sort of “reddiquette” or user code of conduct that emphasizes fair and balanced comments (no sniping). Or, perhaps my “worry” isn’t actually troubling at all. Maybe such a system would be substantially self-policing and refreshing, shifting us from an obsession with ‘perfect papers’ to an understanding that no paper (or review) should be judged on anything but it’s own merits. Given the popularity of pun threads on reddit, i’m not convinced the wholly democratic solution will work. Whatever the result, as with most solutions to scholarly publishing, it seems clear that if PubPeer is to add substantial value to peer review then a critical mass of active users is the crucial missing ingredient.

What do you think? I’d love to hear your thoughts in the comments.