The more specific and deep a community can get in terms of purpose, goals, and experience, the better chance it has of building a large and sticky following.
Of course, it depends on the nature of the topic, targeted audience, and nature of the content, but if your community can add value both to beginners, and to experts, then its more likely to create the kind of engagement it sounds like you are looking for.
I'll give you a few examples:
PE Class vs. Martial Arts dojo
Think of your general PE class at school. Nobody enjoyed doing that for long. Anybody who could get out of it would, or they'd join a sport team and focus just on the team.
Martial arts dojos, however, have a whole program for catering both to beginners and to experts. They understand the maturity pathway a user goes through, and helps them achieve these levels of advancement.
This should be a key design element of your community, in addition to specializing. StackExchange has done this very well with the ability to take a bit of status from one community to another, but mostly keeping the topics separate and specialized.
Apache Software Foundation of Communities vs. a Buffet Style Meetup
This is from my own experience building communities for open source software. If you look at The Apache Software Foundation, it has 284 different projects, which really can be considered very separate communities operating with a common set of standards. Not unlike StackExchange actually. Many of these work very well, and the cross over between communities is relatively small. There is a few "meta communities", but these are people who love the community process even more than specific technology.
A not so good counter example is a Meetup group I run that covers a wide swath of technologies and presentations for different audiences. My attendance isn't nearly as good as a number of specialized Meetup groups that focus on a specific technology, or the needs of a specific kind of member, such as helping Java developers learn their craft.
My end suggestion is to specialize, but get more specific about that specialization. Figure out what audiences you want to cater to. Figure out what their maturity path should be as the progress from aware beginner to engaged expert. How can your community help them get there, and how can you get them to help others along the way.
This will help you create the engagement you are looking for.