I would say, continue what you are doing now.
Respond and interact with the paying clients but ignore (well, perhaps not completely ignore, but have minimal interaction with) the freeloaders. It might be difficult as these people have clearly "bought in" to your product and believe that they have some say in how it develops. The only risk is that these users become so disgruntled that they start posting negative comments about you and your product elsewhere on the internet. You could mitigate this by starting a "wish list" for them to contribute to, but on the strict understanding that you are under no obligation to take up any of the suggestions, or implement them in the way suggested.
If you haven't already, you should probably put a prominent statement to the effect that your project is for your own personal use and for paying clients, but you are not adverse to people taking a fork for their own personal use. Don't let the fact that this was previously "poorly received" deter you, it's there primarily for new users to help manage their expectations of what you can do for them. If these users are truly interested in the product they'll either carry on using it under your terms or fork it.
It might also be worth talking to your existing clients and reminding them that the only person supporting the project is you and while you want to see it grow you want it to be on your own terms and that you can't cope with large influxes of new, non-paying, users.