Peer Review from the Editor’s Perspective

So far, the peer review approach has worked very well for the PHP and Mobile channels. But in some regards, it might still seem like too much work for too little gain. I’ll try and describe the gains more clearly in this text, and address some common questions.

Filter

First, it’s important to realize that as an editor, you’re a filter. If a draft is some muddy water, you’re the filter that makes it drinkable.

There’s no such thing as making water too clean – eventually, you get to “maximum cleanliness”. Adding more filters is just making sure that an unnoticed hole in the main filter doesn’t do as much harm as it could.

Each filter will stop some dirt from passing through which not only makes water cleaner and lets subsequent filters be more effective, but also allows the filterer to better notice the dirt that appears more commonly.

If an author often gets warned by others about spelling / grammar, you’ll know what to expect from them and you’ll know what they need the most help with. On the other hand, if an author frequently makes code mistakes and obviously writes them off the top of his head without testing them in a running environment, that will become noticeable, too, and you’ll be able to warn them.

It’s important to note that in this multi-layered filter approach, you’re still there at the end as the main filter. So even if the “weaker filters” don’t do a good enough job, you’re still ready to clean out the remaining dirt. However, the less strained the main filter is, the more it can be reused – you can focus on other posts almost immediately, feeling less drained.

This multi-layered filtering is by far the biggest improvement peer review offers over the traditional approach.

Lack of People

Some of you are worried about a consistent number of reviewers / authors. You don’t have that many authors, let alone authors who would be willing to do reviews on a regular basis. This needn’t worry you. In our case, peer review is not about finding regular reviewers or a hardcore team that takes care of everything instantly and automatically. It’s about making sure a draft is seen by as many qualified eyes as possible before publication. Whether the person giving it a read is a veteran reviewer or someone who asked to take a look in that instance for the very first time is completely irrelevant – as long as a post is looked at and feedback is given, that’s a win.

In the php-peers repo, I’ve even gotten some PHP influentials to sign up. People who never wrote a single post for us, but care about the community enough to make sure bad resources don’t get published. We often underestimate two things:

The latter is all that needs stroking in some cases – telling someone they’re super awesome for providing some feedback goes a long, long way. Not to mention the attention they get when they voice their opinion in an issue and people recognize them.

Trust

Naturally, this can lead to some trust issues. How do you know the person who asked to join the repo isn’t a spy for a competitor? You don’t. But a stolen topic or two is worth a year of reviewed ones. In time, those who haven’t interacted with the repo in a while will be flagged in the system (under development) and more easily detected and removed (or incentivized to get back into the game), but for now – it’s better to have more interest and trust it less, than to have a 100% controlled environment that doesn’t do much.

Effects of Peer Review

In terms of effects, aside from the fact that the articles are noticeably better (as read on Twitter, Reddit and in the comments of some posts) and get more traffic and shares (I now consider any post that doesn’t reach 10k in its first month a failure), there’s one very important effect we mustn’t neglect.

The peer review system got authors to interact with each other more, fostering a sense of community, belonging and contribution. They feel important, and they want to help, not only each other but the channel and the company – look at the ambassador program for a demonstration of what a feeling of belonging can do.

More concrete editor-benefitting effects include:

Tips

When implementing peer review, things might seem a bit overwhelming. I suggest the following routine:

By following this routine, you implement peer review gradually so it’s not a shock to you or your people, and you get to tweak the process to your liking as you get suggestions from early adopters.


If, through your implementation of the peer review system, you learned something new that might benefit the rest of us, please do share and we’ll expand this document.

Good luck!