Peer Review from the Editor’s Perspective
So far, the peer review approach has worked very well for the PHP and Mobile channels. But in some regards, it might still seem like too much work for too little gain. I’ll try and describe the gains more clearly in this text, and address some common questions.
First, it’s important to realize that as an editor, you’re a filter. If a draft is some muddy water, you’re the filter that makes it drinkable.
There’s no such thing as making water too clean – eventually, you get to “maximum cleanliness”. Adding more filters is just making sure that an unnoticed hole in the main filter doesn’t do as much harm as it could.
Each filter will stop some dirt from passing through which not only makes water cleaner and lets subsequent filters be more effective, but also allows the filterer to better notice the dirt that appears more commonly.
If an author often gets warned by others about spelling / grammar, you’ll know what to expect from them and you’ll know what they need the most help with. On the other hand, if an author frequently makes code mistakes and obviously writes them off the top of his head without testing them in a running environment, that will become noticeable, too, and you’ll be able to warn them.
It’s important to note that in this multi-layered filter approach, you’re still there at the end as the main filter. So even if the “weaker filters” don’t do a good enough job, you’re still ready to clean out the remaining dirt. However, the less strained the main filter is, the more it can be reused – you can focus on other posts almost immediately, feeling less drained.
This multi-layered filtering is by far the biggest improvement peer review offers over the traditional approach.
Lack of People
Some of you are worried about a consistent number of reviewers / authors. You don’t have that many authors, let alone authors who would be willing to do reviews on a regular basis. This needn’t worry you. In our case, peer review is not about finding regular reviewers or a hardcore team that takes care of everything instantly and automatically. It’s about making sure a draft is seen by as many qualified eyes as possible before publication. Whether the person giving it a read is a veteran reviewer or someone who asked to take a look in that instance for the very first time is completely irrelevant – as long as a post is looked at and feedback is given, that’s a win.
In the php-peers repo, I’ve even gotten some PHP influentials to sign up. People who never wrote a single post for us, but care about the community enough to make sure bad resources don’t get published. We often underestimate two things:
- people’s willingness to help the community they belong to
- people’s ego
The latter is all that needs stroking in some cases – telling someone they’re super awesome for providing some feedback goes a long, long way. Not to mention the attention they get when they voice their opinion in an issue and people recognize them.
Naturally, this can lead to some trust issues. How do you know the person who asked to join the repo isn’t a spy for a competitor? You don’t. But a stolen topic or two is worth a year of reviewed ones. In time, those who haven’t interacted with the repo in a while will be flagged in the system (under development) and more easily detected and removed (or incentivized to get back into the game), but for now – it’s better to have more interest and trust it less, than to have a 100% controlled environment that doesn’t do much.
Effects of Peer Review
In terms of effects, aside from the fact that the articles are noticeably better (as read on Twitter, Reddit and in the comments of some posts) and get more traffic and shares (I now consider any post that doesn’t reach 10k in its first month a failure), there’s one very important effect we mustn’t neglect.
The peer review system got authors to interact with each other more, fostering a sense of community, belonging and contribution. They feel important, and they want to help, not only each other but the channel and the company – look at the ambassador program for a demonstration of what a feeling of belonging can do.
More concrete editor-benefitting effects include:
- less time spent warning people about the same mistakes over and over again
- less time spent needing to check the code – at the very least, the common hiccups will be ironed out by the time you get to it
- getting to know your authors, allowing for better profiling. (e.g. due to knowing some of them much better now, I immediately knew who was reliable enough to let in on the PHP Reference project)
When implementing peer review, things might seem a bit overwhelming. I suggest the following routine:
- make a team in the sitepoint-editors organization corresponding to your channel and repo. Mine is
- make a peer review repo
- give the team permissions to read the repo and send PRs
- expand intro docs with your own directives, if necessary. For example, change the min number of peer reviews a post should get before being considered reviewed, or something similar. Mention special programs, if any (I have a regular authors program in which I pay people more as long as they stay very active, and a hot topics program for very interesting topics I want covered ASAP, and also pay more for).
- create initial labels (Editor Review, Peer Review, Language Issues, Discussion, …)
- invite viable authors to the program. Usually, these include very good, very active authors from before, and all new incoming ones. Slowly use this approach to weed out those not interested in having their work reviewed, and use the opportunity to get to know your author pool a bit more.
By following this routine, you implement peer review gradually so it’s not a shock to you or your people, and you get to tweak the process to your liking as you get suggestions from early adopters.
If, through your implementation of the peer review system, you learned something new that might benefit the rest of us, please do share and we’ll expand this document.