How can I measure the quality of the decisions moderators make at scale?

Question

My site is running a user moderation which works pretty good so far. It is very similar to what LoL had implemented with their Tribunal system. A few basic points:

• moderators decide on a post together. So to reach a decision a certain amount of moderators have to cast a vote.
• moderators do not know who is deciding with them and what the others decided (independent and anonymous decisions)
• moderators get appointed automatically based on their engagement, the time spend on the site and their history
• we sort out bad moderators with some security measures I unfortunately can't disclose :(

With this system we could scale quite good and so far we are quite satisfied with the results. But there are users complaining that it is either too harsh or too liberal. It is very dependent on which mods you get when it comes to our decisions (of course not the obvious ones but the more greyish area where the rules are open for interpretation. The problem now is that it is very hard to measure how "good" our moderation decisions are.

It is extremely important for us to measure this in order to improve our moderation system as we scale more.

Right now we have around 3000 daily active moderators that review around 6-9000 posts a day. We currently support 3 different languages with this system.

Here are some options I thought about:

1. Hierarchy of moderation: I could create some sort of hierarchy for moderators where the top ones will be in direct contact with me and then control the decisions of the normal moderators. This sample size then will get me an overview of the health of our moderation system. Problems: I have to make sure that a) the mods I choose are the best ones based on their decisions which I in turn can't all review and don't know the quality of and b) the mods are doing the right decisions by investing time and effort in guiding them
2. Measuring the consistency: Here I would just let some post get reviewed multiple times and measure how often they disagree with each other. This is just a proxy but could still give me valuable insights. Problems: not really sure how to measure the results so it actually gives me some insights without putting too much strain on the system
3. Give users the possibility to challenge a decision: here I would simply give them the tools to challenge the decision and then give it to other moderators to decide upon the post. With this we could measure the amount of challenges and success rate of those challenges. Problem: we are currently only communicating the obvious sure blocks. By showing users the complete range of our decisions without knowing how good they are could lead to a lot of negative effects (hate on mods, lost trust in our system, etc).

How do I measure the quality of the decisions of our moderational system? What experiences do you have with this?

Answer 1

How can I measure the quality of the decisions moderators make at scale?

This a common business aphorism:

What gets measured gets managed

The lesson is that, while you need to measure so you can properly manage, you also have to manage things you can't easily measure.

If you pick just one easily measured metric or another, people will likely game the system to look good on those measures, to the detriment of other, less tangible, goals.

If you can tell, in an automated way, exactly the right decision moderators should take, then let the system handle that so that moderators don't have to.

If you can't tell, your ability to measure is reduced.

The business world attempts to solve this with a Balanced Scorecard.

Balanced Scorecard

A balanced scorecard weights various potentially competing goals, assigning a score to each goal.

These are just ideas, but things you might measure are:

• Ratio of public actions to complaints by trusted users
• Ratio of actions to incorrect actions (maybe really hard to measure at scale, but your voting system might provide a signal here)
• Number of moderator actions
• Number of remediation messages to marginal users
• etc.

And the idea is to assign a score to dimension that you want to measure quality on.

Then you can take various approaches. You could have minimally acceptable standards for each dimension of performance, and if they do not reach those standards, they do not get any credit for the dimension.

Another approach is to simply total the scores from each category (similar to how Stack Overflow moderator elections provide candidate scores.)

In fact, Stack Overflow combines these ideas in the election requirements, insisting on a minimum of meta participation on several dimensions, as well as providing the total score.

Of course, you'll need to apply these ideas to your own system, and not on a one-time-basis, but continuously - perhaps weekly, monthly, quarterly, or yearly - or even some combination.

Conclusion

I think the best way to measure the quality of a lot of decisions is to use the balanced scorecard.

Answer 2

At the kind of scale you've described, you're always going to have some variation. And if it's a worldwide site with participants from many cultures, that complicates things -- I've seen flags I didn't understand because of missing cultural or linguistic knowledge, for example.

I agree with the suggestions in this answer to analyze your data where there were dissenting opinions. I have an additional suggestion for the interface you present to moderators. Give them an "I don't know" (or "skip") option! Give moderators a way to opt out of evaluating something that they don't feel completely confident about handling, and make it clear in your guidance to them that this is not only ok but good. You've got 3000 moderators; if one or two or twenty punt on a post, it's not hindering site operations.

I base this suggestion on two interfaces from Stack Exchange: review queues (where there's a "skip" option) and chat (where there is a "meh" option). Now, chat moderation on SE is kind of challenging, but it's not because of this. (It's because there's a single chat system for (almost) the entire network and sometimes site-specific expertise is required.) Lots of people punt lots of flags, but yet they get handled because lots of people see those flags. I'm not sure exactly how many people have chat-moderation privileges on SE -- at least the 500 moderators across the network, but also a bunch of high-reputation non-moderator users. It's probably not 3000, but it could easily be 1000.

Answer 3

You say that moderator decisions are made independently but they act together. Does that mean that a certain number must agree before an action is taken? If that's the case then you can record the number of times there were dissenting voices. If you record this completely anonymously you'll get figures on how much in agreement the moderators are. If you record a moderator id (perhaps it shouldn't be the same as the moderators' real id) against the decisions, you could see whether there are groups of moderators or just one or two individual moderators who are dissenting from the consensus. If you do record this data then you need to tell the moderators this is going to happen before you implement it.

This does leave you with a dilemma though, if you have one or two moderators who are consistently getting it "wrong" it would be nice to be able to discuss things with them to find out why. You could do this in public in general terms, but you need to be careful that you don't turn it into a witch hunt.

However, you do need to allow for a degree of disagreement between moderators - they're only human after all. There will be cases (like you mention in the comments with "dodgy" jokes, or slightly NFSW images) where one moderator may be happy with it and another not, or the same moderator might take a different approach on different days.

The other thing to look for is if some moderators missed some extra information that led them to make an "incorrect" decision. Was there more context in comments, other posts from the same user, or in their profile information that could have been overlooked?

If you make all moderator decisions available for review by the other moderators and admins, then you can have discussions over the points raised. Moderators can share their tricks and tips and what they look for when making a decision. If all moderators have access to the same information and tools then it's easier for them to make consistent decisions.

Another solution you could investigate is to hold moderator elections, where people nominate themselves (like Stack Exchange) or are nominated by others and then the general user base votes for their favourite candidate(s). You can make the moderator term fixed length if you like rather than the "moderator for life" approach taken by Stack Exchange. If you do, I'd make the term 2 or 3 years and hold elections every year to chose some percentage (e.g. 25%, 1/3) of the moderator team. That way you'll get an overlap in people's terms and some continuity/hand over of knowledge.

Fundamentally though, you have to realise that mistakes will happen. As long as you have the procedures in place to spot and correct these mistakes then you've probably got it covered.