I see a few downsides to a hellban. The first is that it is generally unacknowledged that it exists by the administration team. If trolls knew it existed, it'd be trivial to check for. All they need is a second account (or guest account). If that account can't see the original user any longer, they are banned.
Another downside is that you have a known troll, or group of trolls if you allow this group to see one another, utilizing resources with no gain to the community at large. If banned users can interact with one another, I can easily see they forming their own subgroup that "normal" users can't see. These trolls can still see everything (otherwise it'd be obvious they were banned), but they can make inappropriate comments and none of the regular users can see it. You have a subculture that becomes dedicated to seeing who can say the most offensive thing. It is highly unlikely that the administration team cares or sees this trash without taking the step of entering this subculture.
The process of hellbanning should be permanent. If it's not, when the user is unbanned, they will suddenly have interaction with the community again. Others may notice that the person has returned. Inevitably this leads to some thing similar to:
User: Welcome back @formally_banned_user. How was vacation?
Formally Banned User: Back? I didn't leave. I thought everyone was just ignoring me.
User: Ignore you? Hahaha! You are hilarious. I haven't seen a post from you in a month, I just assumed you'd gone on vacation.
Now what do you do? The concept of the hellban is becoming known to the community and soon you run into the problem when it is just a known fact that your community uses it. Now trolls can check if they are banned.
What can you do about these downsides?
I believe that hellbanning, if implemented, should be the final step in a long list of steps. Additionally, it is not for every user that is banned. If a user need a suspension, then it should be acknowledged (somehow, not necessarily with a public shaming) that a user has been suspended for a period of time. This should be a real suspension - they can't interact with the community at all on your platform. When they return, they can have the above conversation and they will know that they'd been suspended. If the problem with this user continues, another suspension may be in order. This escalation provides the administration team with steps to take to help a user improve their behavior. It doesn't immediately exclude a user from the community. The permanence of a hellban is a huge negative. You can't have users reform their ways if they are forever excluded from the community. Everyone has a bad day/week. Should a bad week exclude you forever?
Spammers don't deserve a hell ban. I, as admin, certainly don't want to wander into the cesspool of hell and see it littered with spam. For one, that means that I've given those bots resources that I could have utilized for other aspects of my community. I want them gone, completely.
