My site is running a user moderation which works pretty good so far. It is very similar to what LoL had implemented with their Tribunal system. A few basic points:
- moderators decide on a post together. So to reach a decision a certain amount of moderators have to cast a vote.
- moderators do not know who is deciding with them and what the others decided (independent and anonymous decisions)
- moderators get appointed automatically based on their engagement, the time spend on the site and their history
- we sort out bad moderators with some security measures I unfortunately can't disclose :(
With this system we could scale quite good and so far we are quite satisfied with the results. But there are users complaining that it is either too harsh or too liberal. It is very dependent on which mods you get when it comes to our decisions (of course not the obvious ones but the more greyish area where the rules are open for interpretation. The problem now is that it is very hard to measure how "good" our moderation decisions are.
It is extremely important for us to measure this in order to improve our moderation system as we scale more.
Right now we have around 3000 daily active moderators that review around 6-9000 posts a day. We currently support 3 different languages with this system.
Here are some options I thought about:
- Hierarchy of moderation: I could create some sort of hierarchy for moderators where the top ones will be in direct contact with me and then control the decisions of the normal moderators. This sample size then will get me an overview of the health of our moderation system. Problems: I have to make sure that a) the mods I choose are the best ones based on their decisions which I in turn can't all review and don't know the quality of and b) the mods are doing the right decisions by investing time and effort in guiding them
- Measuring the consistency: Here I would just let some post get reviewed multiple times and measure how often they disagree with each other. This is just a proxy but could still give me valuable insights. Problems: not really sure how to measure the results so it actually gives me some insights without putting too much strain on the system
- Give users the possibility to challenge a decision: here I would simply give them the tools to challenge the decision and then give it to other moderators to decide upon the post. With this we could measure the amount of challenges and success rate of those challenges. Problem: we are currently only communicating the obvious sure blocks. By showing users the complete range of our decisions without knowing how good they are could lead to a lot of negative effects (hate on mods, lost trust in our system, etc).
How do I measure the quality of the decisions of our moderational system? What experiences do you have with this?