Braiam is saying right that it's important to 1) set ground rules early 2) let everyone know exactly what happens if the rules are broken. However:
It is better to lose a user who brings harm, even if it means losing some benefits he was bringing, because even one disruptive user can create huge problems in your community. Read about the no asshole rule if you want to learn more.
On the other hand, punishments are often not specified exactly in small communities, specific set punishments for specific rule violations are more of a feature of big communities, because...
- ...in a small community you have more time for each of your users and you can afford to educate them on how to behave. You can explain things a lot of times, read long unban appeals, give 2d, 3d and 4th chances, do this all by yourself, and still have a life. In a big community you don't have this opportunity, so a user soaking up too much attention is removed once upon a time. So, use your opportunity to its best.
Monica Cellio has brought up an important question of the balance point between not allowing an asshole to disrupt your environment and being too strict. Well, it is a bit opinion-based, but here are some guidelines.
- Never let their junk stay. Have the problem messages deleted, edited, or whatever is considered OK for your community. Be clear that it's not a punishment, it's just cleaning their junk, that it's not something you want to do, but rather are forced to do.
- Give an opportunity to learn and reward their wish to learn with additional chances to try. Explain 100 times if the user really doesn't understand and tries to. Punish if you think that a punishment would help, be creative. You can just use short-term suspensions, but creative, case-specific punishments are on average more effective. See appendix for an example of a game-specific punishment of a gaming community I was in.
- While the user is learning, it is better to minimize potential harm the user may do. For example, if a really problematic user on your forum is learning, turn pre-moderation on for that user.
- However, if you see that the user really, really doesn't seem to learn, it is likely better to give them a suspension long enough to change. Since my experience mainly comes from gaming communities that had many teenagers, there was barely any point in banning someone for more than half a year: after that time the problem user often changes enough to be treated like a new person. The best suspension time I have seen in use was "until appeal" -- basically, until the problem user shows that he is capable of writing an essay and able to understand what exactly was wrong. If you have lust, you may keep some private communication with the problem user to explain something further, but my experience shows that it doesn't help unless you see that the user really did change their attitude. You can just set a minimum time when you will be ready to read their PMs and unban appeal again so they calm down. If you community is small enough for such stories to stand out, write a letter to that user (better privately), stating calmly that you do not consider the user to be a good fit for your community for the following reasons, and that you are suspending them until they show the ability to act maturely. Tell when are you ready to read their first unban appeal.
- Do not mistaken the statements above for being ready to keep a user who actually creates disruption. If real disruption still appears, the user "really doesn't learn" and has to leave. Only reward the user with additional chances if you see an overall positive pattern and are ready to invest time. If the pattern is not positive or you don't want to invest time, don't bother. No matter the patterns, do not allow the user to cause damage.
Appendix: an example of a creative punishment.
Persistent World is a modification for a game called Mount&Blade: Warband. The original game's multiplayer is a slashing simulator, the mod focuses on role-playing. Because of very special nature of this mod, most of the servers that used it were gaming communities with relatively long rules that couldn't be either forced by software solutions or specified strictly enough to avoid any kind of loopholing.
There were several types of common offenses: Random Death-Matching (attacking anyone on sight, is bad as it denies the person an ability to role-play with you) and New Life Rule/No Revenge Rule (when you respawn after death, you are not allowed to provoke someone who just killed you, is bad cause it makes your death cost nothing), being the two primary "minor offenses", and Combat Logging, being the most harsh offence, logging off during combat to avoid character death and keep valuables that your character has. It denies people who were going to win the opportunity to savour the victory, it is very frustrating to see it.
For Random Death-Matching or No Revenge Rule offenses, provided that it happens for the first time, a typical punishment was a 15-minutes talk with an admin about the nature of those rules (notice that the admin is supposed to have those 15 minutes!) and a warning, or slaying of your playing character (he can respawn in 30 seconds to play further, it's just supposed to be a bit frustrating). If you do it again, unless you begin a server-size massacre and really disrupt roleplay for everyone and don't seem to learn, you can get permabanned, but normally the biggest time you can escalate to is around 30 seconds.
For combat-logging it's way stricter. First, you do it to save your valuables illegally, right? You lose those valuables if you are caught. They are deleted or given tho those who were going to get them if you didn't combatlog. Then you are banned. First time offense is typically a one-day ban, then 3 days, then for 2 weeks. If you commit it 3d time, you lose the ability to save your valuables when you log out unless you do it in a safe zone. It means that a normal user can suddenly decide that he/she is sleepy, log off, go to bed, sleep, and log on next day with all his/her things, but you have to take a long route to a special area where you have to log off, or your things will drop on the ground when you log on again.
This is announced openly, not something applied silently like shadowban on reddit, users don't want that, hence combatlogging is very rarely seen among established users.