The biggest problem in doing this is that there is nearly no way to automate things, meaning that you will need to do a lot of things manually and spend time monitoring the competitors. Some companies even have dedicated personnel to watch the market and companies evolving around it.
I will cite an article from kissmetrics, you can read more about it here (comments in brackets from me):
Are these sites [Alexa, Compete, DoubleClick and Google Trends] 100% accurate? No, they’re not. But they all offer some good data that you can use for competitor research. You can easily use them to do the following:
- Compare traffic – although the numbers might not be accurate, you can still see whether Site A is getting more traffic than Site B on average.
- Learn more about their keywords – most of the above mentioned networks share some basic keyword data. You can use this to find out what the competitors are targeting and get some keyword ideas for your own website.
- See related sites – find additional competitors by looking at related sites and add those to your research.
This will already give a good idea on how the competition is doing and you already figured that out yourself. But in order to measure growth, you should also take a very good look at the page and the community. Both things can give you quite the insight about how big and resourceful a community is. But before doing that, I will tell you some things about data and obtaining it.
Best thing upfront, most sites will share it quite willingly, sometimes even post it straight into your face so you are surprised that their product / community is so amazing. So, prepare your Excel worksheet and grab the necessary data.
If communities are run on a forum software, your best bet is to scroll right down to the page and take a look at the site's statistics. bbPress, phpBB, vBulletin and many other softwares are already configured that way so that they post the latest statistics right at the bottom of the forum page. You can then scroll down and get a grasp of the overall userbase, today's posts and threads and the newest user. If the last one changes frequently, you can be sure that the community is growing at a steady pace. You get your Excel worksheet and connect it to the necessary data - that way you can pull it daily, bi-weekly, weekly, monthly, however you like it. Right after that, you can draw graph with the data and analyze it as you have the userbase of everytime you pulled the data and the day's posts and threads. That way, you can even see if there is a drop or increase in engagement and activity.
As most smaller communities run on commercially available forum software, you have some very good data at hand, provided by your competitor. You can even dig more deeply into it and search the whole forum for hot posts and threads, most engaged users and so on. However, more advanced search tools often are a privilege for members of the community.
Bigger communities that run on forum software usually have custom built software. This can help you even more or make it a hassle to grab any data. As an example: StackExchange makes it easy. I have searched our community with two searches and I have gained some very precious data.
I searched the community for the last month, August, to determine how much content is created and how active the community was:
Doing this on a big site bi-weekly could provide you with a lot of information. Combine it the information gained from Alexa and Google Trends and you can view correlation between data and might even be able to predict the near-future trend and growth of the site.
So essentially, data can be mostly obtained from the sites themselves. This can help you quite a lot. If it's not provided by the site, you must fall back to some visual observations.
A good idea to monitor growth is to take a very close look at tags and / or categories in a big community. If tags and / or categories are added, you can be sure that the community is having or expecting a significant increase in this particular field and need to sort the data. If the community doesn't have an apparent organization and misses categories, it either has its own method or is experiencing a massive surge of inactivity.
Another way is to look at the webpage itself. If you can recognize familiar structures or other commercially available software, you can be sure that the community isn't resourceful enough to build own software. This translates into that it doesn't have the resources or doesn't see the need to have a custom-built software. This can furthermore relate to a medium-sized community that isn't growing exponentially but linearly. And the linear growth is at a steady pace, nothing too bad, nothing too good.
However, if you see custom-built software, it's sure that commercially available software just wasn't enough. This relates to a very demanding and probably big community as the standard solution doesn't fit anymore.
Yet another hint can be sudden maintenance and or sudden changes of the software or general overhauls. This means that the community is having or expecting a growth in traffic and needs to prepare or adapt right now.
Note that these optical and visual hints aren't reliable, oftentimes you need to confirm them by either signing up and viewing the community from the inside to see if it's really busy or to, for example, read their blog to get an update in their latest changes. (Everyone would announce a big change.)
The final summary is that you quite often can necessary data from the sites themselves as some sites just post it straight away. This is good for you and you should use this opportunity. If they don't provide data (the userbase and the total posts / questions count are usually available) , you need to rely on Alexa and other services and on first-hand experiences from yourself: how organized the community is, how many changes are made when and why, "standard" software versus custom-built one, and so on.