The best bet is to meet the needs of the community. On the one hand, you have a community focused on a particular topic, on the other hand, you have a subset of that community interest in other topics. Both groups are part of your overall community. People get to know each other and enjoy socializing, even if it doesn't fit the bounds of the topic that originally brought them together. Online, there isn't an option for somewhere else to hang out with people unless someone provides one.
If you don't provide it or try to block it, eventually, people will leave to have the socialization they want and that may make them move their original participation in addition to the newly forming community. If you want to avoid this, provide a place for them to be able to have the fledgling community. Create another associated channel. This can be a general off topic or general channel if the users just talk about life or it can be one or more other topical channels if the users frequently talk about the same thing.
This does put a little more burden on moderating the community and it may still take some pushing to get users to use the proper room, but it allows them to continue to participate with their friends in the community the way they want to while allowing them to remain within your overall community.
This is a large part of the reason behind StackExchange chat allowing almost any user to create a new chat room for a discussion that doesn't fit within the main chat, though IRC unfortunately lacks the same room hierarchy features. You might be able to kind of replicate it using naming conventions though. Have users create channels like #communityName-topic, assuming that the IRC network doesn't have a problem with the possible proliferation of channels, particularly if they aren't permanent channels.
If you can get /fjoin permissions, that can be a helpful ability to use for pushing people in to the channel that they should be in, but not all networks are going to give out /f functions to a channel operator.