I have to disappoint you.
You're asking the question regarding building up an audience. A lot of streamer and people who operate in the same business, e.g. youtuber, have the exact same troubles as you have. I came across a lot of people asking how to grow their sub count. This is tough, and most likely you don't like the answer. However, I'll tell you about the things which can be done to influence your growth positively first.
Be consistent and regular. Elaborate clear streaming times, and arrange your time schedule so you are available to these times. Nobody likes to see that the stream or video which are awaited so long just aren't online to these times you set up earlier on. Also, your audience sees it as a positive step: You value your community so much that you created space for it, and even on a regular basis.
Be random. I know, this seems to be ridiculously contrary to the mentioned regularity. Imagine you are a regular spectator of a stream; it can be long to wait for the stream. Lose up your spectators by providing some random streams but be clear about this one point: It's a random stream, not a regular one. You did it once in the week, this doesn't mean you do it again. If you keep spamming random streams, it can hurt your relation to your spectators. Even though you have regular times, spectators most likely feel like missing something, the information from your other random streams. As these informations aren't available for everyone (because you didn't set a time), they will ask you to set a time. That's not gonna happen because it's a stream which is provided when you feel like it. You can provide random streams, but you shouldn't overuse it.
Give back the love. People subscribe to you to not miss anything you do. Although you already stream, you can get your interaction on a whole other level. Play with your subscribers, do a giveaway. Your subscribers will feel like really being part of your community as everyone has the chance to play with you. Also, mention donations. There's this feature on twitch which can show the latest donation, enable it. So they have the opportunity to donate as well as reach out to you and your community.
Be creative and unique. Don't do the trendy things, expect if you want to build up a "common" community. You don't want a common one. You want a community which helps you out and feels like being a part of your gaming life. Pewdiepie refers to his subscribers as "Bros", he created a synonym for the word itself -- he let the people grow closer. You are the streamer, you build up the community which is built to be about your gaming life and your streaming. Shape your community with the things you take a liking to.
Be fair and neutral. This probably is the hardest thing to achieve. You'll come across trolls and haters, earlier or later, it doesn't matter. Even if this one long-term user who decided to flame and blame you so excessively is one you like, you have to kick him out. Otherwise your subscribers think that you value some members of your community more than others, this will harm your growth. The group you apparently don't care about that much would feel betrayed -- they would simply leave.
Look for a partner. You can "exchange" subscribers with another stream. Just do a collaboration and let both communities meet, of course you could some subscribers during this step too, but you can win some as well. It's all about getting your name out there; doing collaborations or creating advertisements will help you with this.
Use social media. Granting a perspective in your private life is a risky step but it can be rewarded. By creating a fan page on Facebook or Twitter and publishing some stories about your life, you can attract more people. Even if your life couldn't be more boring, there are people who are interested in your eating and sleeping rhythm. Of course, you have to limit yourself -- don't publish anything that could be used against you. The internet doesn't forget as your community.
Now we'll head to the disappointing part. Even if you do everything right, you are available, you are open to your community, you can still be not able to succeed. The chances are against you. The biggest factor in community growth is luck. I could list you a lot of start-ups which weren't able to attract customers, they had to reboot several times.
Administering and moderating a community is about your actions, how you interact with your community, how you reward, how you punish. Growth of a community has two factors: your actions and the luck of being discovered. And luck is the heavier factor here.
One positive thing in the end: You'll grow eventually, even if you don't show qualitative videos or streams, there still are people who are attracted to lower quality. You'll grow in the end. How fast is bound to the results of your actions and the odds. And the best thing: As you're getting more and more famous, they factor of luck is steadily decreasing. By getting your name out there, you aren't too depended on luck anymore.
