Be careful what you wish for
Peer Review Monster Gideon Burton
Some say peer review of research articles is broken and needs to be changed, others that peer review needs a killer app that allows readers within a social network to vote a paper up or down. Cameron Neylon says Don’t (peer review) – the risks outweigh the benefits of publication. He says “publishing anything does no harm’ whereas peer review is costly in expert time.
Before we throw the baby out with the bath water, let’s think about what’s good about peer review and what could be improved.
Peer review can be good when:
- it offers blind review when reviewers do not know gender/age/race/status of authors (unless this is revealed by text and context)
- reviewers offer constructive advice for authors to improve their article/ creative work
- it winnows submissions to journal (or other publication outlet) to improve quality of what is presented and filters content for readers
Peer review can be bad when;
- it introduces delay in publication of work (tardy editors/ reviewers, backlog of accepted articles)
- opinionated reviewers destroy confidence of authors
Alternatives to ‘traditional’ blind (author does not see identity of reviewer) or double-blind (neither author or reviewer see each others’ identity) reviewing include open peer review and the various recommender systems such as reviews on Amazon.com, Facebook ‘likes’, etc.
What are the risks of abandoning peer review ?
Well of course, the short answer to that is “I don’t know” but a search of scholar.google.co.uk reveals that it is a topic worthy of discussion.
“The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose ‘invisible hand’ is what maintains its quality,” Steven J.Harnad from Peer Review : A critical inquiry by David Shatz.
What concerns me is that abandoning peer review will leave us more exposed to the gender, race and other identity biases that remain with us, and flourish in online communication where identity is key.
What do you think?
Do the benefits of abandoning peer review outweigh the disadvantages, and why?
PatParslow July 23, 2011 - 7:54 pm
I agree on the potential problems with various types of discrimination causing problems, but that is something that well managed pseudonymity can deal with in open peer review, I think.
Sadly it seems that peer reviewers seldom provide constructive criticism (indeed, the evidence I have seen rather indicates that they seldom read the paper in question and just use the opportunity to put down any attempts to publish on a topic – sometimes publishing the same thing themselves not long afterwards).
Winnowing submissions is probably not a positive thing. Again, this is a personal view, but I see very little to indicate an improvement in quality in peer-reviewed publications over blog posts (although I do admit there are worse cases of rubbish being published in a totally open model than in a peer-reviewed one).
Delay is a bad thing, and coupled with institutional requirements, it can mean that submitting a paper to a journal which peer-reviews can mean the work is essentially redundant by the time it reaches the public, if, indeed, they are ever able to read it without spending a small fortune to do so. On top of that, delays in the process can be career limiting. For instance, a requirement to publish X papers during a PhD, coupled with time-to-paper of around 12 months (or more in some cases) means that many PhD students will only be able to publish very early research within the time span of their studies.
In some fields, however, a rigorous peer-review process is probably still warranted. Unfortunately, some of the best examples of the process failing to support quality are in those same fields – Medicine, for example.
francesbell July 24, 2011 - 9:57 am
Pat, I am sorry that you have had such a bad experience of peer review. As an editor, I work hard to facilitate constructive criticism (indeed this can contribute to delay whilst I look for ‘better’ reviewers when initial reviews come in). If an author saw their work published after being reviewed in our journal, I would certainly want to know about it and try to pursue it.
The question you raise about PhDs is an interesting one. I don’t know of publication being a requirement for PhD but I do think it is probably more difficult to fail a PhD, part of which has been published in peer review outlet.
Lastly, I am very excited that Research in Learning Technology is going Open Access from 1 Jan 2012, and hope we increase our readership and other impacts.http://www.alt.ac.uk/publications-and-resources/publications/alt-journal-research-learning-technology
timjohnson July 27, 2011 - 8:52 pm
Sorry Frances, I have to agree with Pat. There is far too much abuse of the “traditional” system of peer review. I think that far worse though is the way that good research is not published, published too late or hidden in very expensive academic or professional journals. Young researchers very often do not stand a chance just at the very time when they need to get their names noticed.
Why can we not all change to a system like PLoS ONE ( http://www.plosone.org/home.action )? They publish about 70% of the articles sent to them (and they are proud of it). Unlike many journals which actively try to limit the number of articles printed just to make them appear more worthy.
This is a tremendously large and important issue we really need more opportunity to discuss it.
I am so glad there are more and more journals that are Open Access and that Research in Learning Technology will be joining them – just take it one step further and change the way articles are peer reviewed
francesbell July 27, 2011 - 10:12 pm
At Research in Learning Technology, we are still in the happy position of deciding on quality not by limiting numbers, and we certainly don’t reject to make ourselves ‘look better’.
PLoS One charge a publication fee of US$1350 (waived under certain circumstances). ALT fund publication out of membership fees, and though I can’t speak on behalf of ALT, I am pleased to be moving to Open Access so that the investment of members, authors, reviewers and editors can be realised by reaching more readers.
You ask us to change the way we peer review articles but I am not sure that our approach is so very different from PLosOne “PLoS ONE will rigorously peer-review your submissions and publish all papers that are judged to be technically sound”. Where we differ is that we consider the Aims and Scope of the journal, and since our members are paying for it that seems to me to be entirely reasonable.
There are several different economic models for publication but they all ‘cost’ in concrete financial terms and in time costs for authors, reviewers, and editors (she says with feeling 😉 ).
timjohnson July 28, 2011 - 9:55 am
Sorry if I sounded like I was critisizing RTL in anyway – didn’t intend to.
Yes I know that someone has to pay somewhere at some point and that PLoSOne do it by charging the author. I was trying to say that this is a bigger issue. I think it relates to what we believe about knowledge and wisdom, who if anyone has a monopoly on them and how they should be shared. I agree there should be some peer review but not in the sense that it is censorship. I prefer people to be able to say they like or value a piece of research rather than being told by some “higher power” that this is what you should read.
I feel very old and jaded, and exasperated. I’m so tired of all the silly behave over copyright and the “reluctance”/”inability” of academic publishers to produce ebooks that are not just PDF. I really did have higher expectations of the human race – silly me 🙂