It's often difficult to detect a genuine troll from a confused user. As @AffableGeek stated it's best to give the user the benefit of the doubt at the start and treat them as if they are a new user who simply isn't aware of the rules or how things are done on the forum.
One of the best ways to catch a troll is to watch for trends and to use the tools available in a community. With a combination of the following you can normally catch trolls out:
- User notes: mostMost communities have some way to keep notes on a user. Moderators can use these to keep track of warnings / notices handed out to users suspected of being trolls. If several threads / messages attract similar notices then you have a pattern of behaviour.
- History: By going through a usersuser's history you can check for a trend or pattern of behaviour. Has the user done this before? Do they make a habit of inflaming comments?
- Private communications: trollsTrolls seek attention, so taking things private denies them their audience. How do they respond to private messages asking them to behave and respect the community rules?
- Penalties: lotsLots of communities will penalise a person with restrictions and even temporary bans before permanently banning someone. Has this person been subject to these restrictions in the past? Have they responded to them?
- Other moderators: whatWhat do your pierspeers think? Discuss things with other moderators and get their feedback too. Human discussion is often invaluable in identifying a troll.