This question is about a Stack Exchange site, but may also apply to other sites with similar voting mechanisms.
Stack Exchange, for better or worse, allows anonymous downvoting. People are strongly encouraged to downvote incorrect or useless content, but users are ultimately left to decide how they want to vote.
Now, in this particular community, we have a couple of users that are prolific anonymous downvoters. They are very active voters and they end up downvoting roughly 20-25% of the total posts on the site (questions and answers - it's a small site). Sometimes it's easy to see why the post warranted a downvote, but other times it's totally confusing. I see well-written, correct answers downvoted on researched questions, with no other competing answer on the question. It's also not a case of serial downvoting, because the downvotes are spread out across a large user base.
Where this becomes a problem is that the experts on the site - our most prolific answerers and top rep users - get discouraged from posting because they will very frequently be hit with multiple downvotes a day. We've now lost about 80% of our best answerers and a few others have complained they're getting tired of the downvotes and want to leave the site.
So, from a community standpoint, these drive-by downvotes have damaged the core of the community. Yes, new people come in to fill the answer gap, but then they too get fed up, and eventually leave. I've now seen this pattern play out for several waves of users across multiple years.
Technically, I don't think the downvoters are violating any rules, but I'm worried about the future health of the community. The community has in effect gained a negative/hostile reputation and is having a harder time attracting new users and new content.
The downvoters also have not voluntarily offered reasoning behind their votes, which they are allowed to decline, but makes it difficult to understand their perspective. I am also worried that if they revealed themselves, there would be a very messy and very public pile-on.
How can I resolve this conflict?