How can I tell if it's an actual problem and it makes sense to approach it, as opposed to leaving it as it is?
I would start considering whether the conversations you're having that are on-topic in your off-topic chat room are worth being re-posted or newly brought up in the main chat room. If they are, then you might want to try approaching it a little more formally.
If not, then you might be envisioning a problem that isn't really there.
Is it a common phenomenon?
Absolutely! I can speak from personal experience in this regard.
When I create a splinter chat room from a larger community's group, often times it has a lot more activity than the main chat room as it feels a lot more relaxed. However, if we're speaking of something that pertains to the larger community, I try to ensure it's brought up with them in an easily-digestible manner.
The fact that you recognize that these topics are on-topic for the main chat room in the first place is really awesome. Most people don't think twice about this.
What are the possible solutions, and how could they backfire?
I want to go ahead and say that keeping discourse on-topic and in the right place is an uphill battle. There is no absolute, unbreakable solution. Especially when it comes to loosely-ruled, friendly chat rooms.
I have a couple of ideas for you.
Move or Re-Post the On-Topic Discussion Topics in the On-Topic Chat Room
Re-posting this material (with the consent of the original author) has the benefit of bringing that conversation directly into the limelight without much "moderation" necessary. It might even be worthwhile to do this on a case-by-case basis. Perhaps one topic of discussion was really noteworthy and in need of deeper discussion. Similarly, perhaps another topic of discussion was barely on-topic and further discussion wouldn't be too entertaining/helpful.
Moving the discussion should encourage your other members from the off-topic chat to participate in the main chat room (and if they don't, maybe give them some encouragement?) Soon enough, I'm sure the discussion will continue right where it left off, but with a host of new voices able to contribute.
If you find yourself doing this often, then I would try encouraging your off-topic chat room members to really try posting on-topic discussion in the correct chat room. You don't have to do this from a managerial or moderator perspective by any means, merely as a fellow chatter who wants to include the rest of your community members in discourse. Try to frame it from a "Hey, this conversation is actually going somewhere, we should see if anyone in the on-topic chat has any input on something like this." perspective.
The downsides of this approach arise when you realize that you're not always going to be there to try and sway other users to post in the correct place. Additionally, not everybody is going to comply, as they might not really want the input of the other members. That's something you're going to have to handle delicately.
Try Inviting Members from the Main Chat to the Off-Topic Chat
Perhaps they want to be involved! If they don't, that's okay, but including the other members does foster a sense of... well... inclusion, and it ensures that they don't miss a thing.
The downside to this approach is that not all members in the main chat room are going to want to see your "sh*tposts." Using this method in conjunction with the first method (moving on-topic discussion into the main chat room) should be an okay solution.
So far, I decided to write a message on our mailing list where I'm trying to gather opinions and suggesting to modify our bot so that it reminds people to consider moving to the main channel if one of the on-topic keywords is mentioned.
That's a really smart solution. I would try to ensure that the bot's message is as nudging it can be without being authoritarian, if that makes any sense.
Thing is, I'm a bit anxious about over-regulating our activity and harming the community as a result.
As you should be. That's one of the hardest parts about trying to manage a loosely-knitted community without placing yourself in a moderation seat. You want everyone to feel equal, but you also want them to follow some rules. In this case, I would try to lead or guide your fellow members into the right direction, whether it be by carefully worded influence, or by example.
I would try to avoid commands or other forms of acting with authority. For example...
Don't say:
"That doesn't belong here, post it in (main channel name.)"
Instead, say:
"Hey, that's something I think the people in (main channel name) could have more input on. Do you mind if I re-post it there, or would you like to?"
The former statement is an obvious "Do what I say" statement, whereas the latter is encouraging further discussion without any form of authority interjection. Even if the response is simply, "Nah, I'm not very interested in the topic anymore, I think we've exhausted it," it's still worthwhile to try the gentle approach.
