Ask the users! Specifically in the following categories:
- Ask the people who are leaving / have left
- Ask users for feedback with their moderation experience
Both these ideas come from World of Warcraft, who have the most well regarded moderation team I've ever heard of.
If your game has a "Close account" feature, or you don't but you have a way to monitor which users disappear for long periods at a time, ask them why. Give them a simple form to let them explain why. For example:
We're sorry to see you go, to help us improve the experience of our future users, please tell us why you are leaving:
- No longer enjoying the game
- Issues with another user
- Issues with a moderator
- Financial concerns (if your MMO isn't free)
- Other, please specify (Text area below)
If they have a problem with a user or moderator, ask them who and review their interactions. If they give a specific reason you can also search the field for other usernames mentioned and do the same thing.
Just asking them what the problem is tells the user that you care about their experience and that you may be able to do something about it.
Whenever you interact with a Game Master (moderator) or customer support agent, you are afterwards asked to give confidential feedback about the experience, such as asking if you think the problem is resolved, how you'd rate their professionalism and knowledge and any other comments.
This information is then used to determine how to carry on. If you gave the moderator a low rating and said the problem isn't solved, you will then be contacted by a different moderator who can follow up on the issue as well as ensure the previous moderator handled the situation appropriately.
This whole system lets the user know their feedback is important, that all moderator actions have oversight and removes a lot of guess work from your user action analysis.