I'm pretty sure we will never know of a satisfying answer as this is a question every community manager or leader is concerned about. I'm also pretty sure that my answer won't satisfy you.
Let's start with something really basic, at least something basic to me. I think it's highly unethical to judge your community by using numbers. Numbers are related to economics, and if you rate or measure the potential of your community, you just want to know if it can compete against something or if it's capable of buying stuff. De facto, there's no universal scale which will tell you when a community is good or bad and there will never be because humans are unpredictable.
Let's talk about the Community Health Index of Lithium. The first thing you see in the table of contents is the "Executive Summary".
In the current economic climate, companies are discovering that their online
communities have become a powerful and cost-effective vehicle for interacting with
customers. For example, a consumer electronics community that runs on the Lithium
platform recently reported 1.4 million deflected support calls, resulting in an annual estimated savings of $10 million.
From this point on, I can pretty sure say that the Index is a business approach. You have a community? Fine, Lithium is gonna tell you how to earn money with it or how to save money with it. It makes sense to measure a community in numbers to analyze its economic potency, to figure out if it can compete with other communities in the same field. The Index will definitely help you to find out what economical direction your community is going but it won't tell you where it's going emotionally and socially since this isn't a big enough value in the newly discovered term of Online Communities.
If you discover something new, corporates will try to figure out how to make money out of it. It's normal to see something like this, and they find a way to do it eventually, nevertheless they will ignore other facts for the sake of money. I will explain this at the "measurements" of Lithium.
- Members
- Few or many members say nothing about problems. It means that you're community is still growing or that the interest your community is built on generally has a small or large audience. You can't calculate anything with the member value as it says nothing about the feelings of the community.
- Traffic
- It's typical. Low traffic, less opportunities to promote stuff, higher traffic, more opportunities to promote stuff. Traffic is a number which also doesn't help to find problems and possible solutions.
- Content utility (a function of the number of posts and the views)
- "How many views did you get on your last article?! Nevermind my question, it's too low, anyway as you can always increase it!" An open-ended way of measurement.
- Responsiveness (how fast does a post get interactions)
- Finally something interesting as it shows if users appreciate your community that much that they try to keep up to date all the time.
- Interactions (unique visitor participators and comments)
- "How many interactions did you get on your last article?! Nevermind my question, it's too low, anyway as you can always increase it!" An open-ended way of measurement.
- Liveliness (how much activity)
- Quite broad. Perhaps, it can help to find some spots.
We see that at least four of the six ways of measurement of the Community Health Index are thought to measure your economical strength. Not many members? Lithium isn't interested. Not enough traffic? Lithium isn't interested. Not enough content utility? Lithium isn't interested. Not many interactions? Lithium isn't interested.
There's not this way of measuring your community socially, economically, culturally and emotionally. If there was, politicians jobs would just have gone. These people try to manage the largest communities of the worlds, also known as nations. I'm not a politician but I'm sure that such an Index might not work.
Summarizing this all, everything I can tell you out of an ethical point of view is to get into contact with your community as often as possible and as personally as possible. If you want to know if your community is struggling with anything, just ask them, they will complain earlier or later.
Let's get to the more economical approach of this answer: I'll try to modify the Index a little bit and add some values and ways of "measurement", however, please don't forget that I'm not paid to do so and that the development of the CHI took some time.
Everything of the current CHI plus some own points:
- Members
- Traffic
- Content utility
- Responsiveness
- Interactions
- Liveliness
- Ratio of posts and reports
- Obviously, it's important to keep this tracked. If it rises, your rules are too loose or may be easy to bypass. It creates unrest in your communities as low quality content keeps popping out. This will hurt your growth a lot!
- Ratio of posts to members
- Do you attract a lot of new users or old, and known ones? If it's the latter, the entry barrier might be too high as the same users keep posting and new users just don't get the hang of something specific.
- The time it takes to register / sign up
- The more it rises, the more hassle it is to sign up. Do you want to spend 10 minutes just to get access to this one post?!
- The availability of staff
- Staff has to be reachable and available at almost every time. The moment people aren't able to contact you, they will lose interest and think you don't care enough. Instant gratification is important in this time and age. Generation Y doesn't like to wait for you, they will just leave then.
- The length of posts
- Longer posts indicate more engagement, shorter posts indicate casual conversations. This helps you to determine if your community is fast-paced or slow-paced. Slow-paced communities usually are built up on the interests of politics, science and everything that needs some time to understand it. Fast-paced communities usually are built upon celebrities, games or movies.
I added these four because I think that they are a stable way to "measure problems". Of course, I couldn't conduct any research and these points may turn out false and untrue. You should take into account Generation Y. They are valuable customers, they will buy almost everything if you can deliver on time. A lot of things are about time, and you know the proverb "time is money". Every second you lose in cluttered UX or in a pointless discussion with staff is lost investition time.
However, I want to remind you that I basically think the same as @AnnonomusPenguin: There's no automated way to measure your community. While I don't think it's random how "high the quality" of your community is, I completely agree. If you really want to know the health of your community, ask them. They will tell your better than any number could ever do. What do politicians? They hold polls to get the opinion of the people, they ask them what they want to see changed. A number can tell you a lot, a member can tell you even more.
