How does a mailing list compare with a web forum in terms of user engagement and retention?

Question

A discussion-oriented mailing list I am (was?)¹ on has had a recurring discussion about moving to a web-based forum. Those who favor the move argue that content is easier to manage, presentation is nicer, and mailing lists are so 20th-century. (Somebody usually brings up threaded conversations too, though many mail clients also support that.) Those who oppose it argue that push (send me mail) and pull (I'll go to a web site to look) are fundamentally different, that an inbox is easier to manage than keeping track of what you've read on a web site, and what's wrong with good ideas from a prior century?

Both sides argue that their preferred platform is better for keeping users and encouraging participation. But so far it's just assertions -- nobody has any concrete data to point to.

So my question is: How do these two platforms compare in terms of user retention and participation? What data is out there and what does it say, so that moderators can make informed decisions about these platforms?

_{¹ This particular list has just made this change; the mail host was shutting down and the moderators decided to use that opportunity to take the plunge. It's too soon to say what it means for this community, but this topic comes up on other mailing lists too, so the answer will still help me.}

Answer 1

Disclaimer: The following answer is the result of a few days' investigation by a non-expert in the field, drawing entirely on publicly available sources. It should not be taken as a comprehensive review of the literature.

This question piqued my interest but I don't know of any data or studies that address the very specific comparison you're asking about. There's quite a lot of research out there on web communities but those that look at empirical data tend to focus on one individual platform. I saw a fair number of articles that drew entirely from Usenet, which isn't totally irrelevant but doesn't capture the range of what a modern web forum provides.

The right technology for you

One article from the University of Haifa^[1] seems to argue that the compatibility between a particular technology and its user base is more important than the choice of technology itself. The authors say that a taxonomy of "virtual publics" should address: the subject of conversation; the geographic and demographic characteristics of the user base; the social structure of the community; and the discourse architecture, which is essentially the term they use to refer to the platform and its underlying technology.

To achieve [their] goals the managers of virtual publics need to ensure that the virtual public's discourse architecture enables/encourages the discourse form desired. (218)

The article points to one main cost of participation in "virtual publics" being the stress users experience as a result of a communication-processing load. The more content they have to read and process in order to participate, the greater the stress. Too much stress leads to information overload, but the consequence of which is only vaguely stated; "beyond a particular communication-processing load, the behavioral stress zones encountered will make group communication unsustainable" (218-9).

The claim here is not that technology will determine online behavior, rather, that technology is both an enabler and constrainer of actions. Each class of [computer-mediated communication tools] will have its own associated stress zones that can be measured. For each class of CMC-tools, it is also likely that the point at which information overload becomes an issue will relate to the nature of the virtual public's discourse. For example, virtual publics which aim to support empathetic communities ... may require a greater level of interactivity and thread depth to sustain appropriate discourse than a virtual public focused on software support. (220)

Information overload

The practical response is to ask, well, what's the consequence of information overload? Does it tend to cause users to participate less, or to disengage entirely? Is it simply a pressure that imposes a carrying capacity on the community, or is it something catastrophic that could cause a community to collapse? These questions are raised in a follow-up paper from the same researchers^[2]:

Individuals can take a range of actions to reduce the impact of information overload resulting from group CMC.... [which] can be combined into two primary options for a population of experienced users who cannot easily adopt new information management techniques. The first option is simply to end participation. The second option is to change ongoing communicative behavior.

The main point I take away is that either type of platform could be better at engaging and retaining users, depending on the nature of the user base and the goals of the community. This seems to fit with what I see in web communities in that certain demographics are much more comfortable with certain technologies, especially with respect to age.

The other key point is that information overload reduces engagement and retention. So in choosing between platforms, ask yourself how each option might allow your users to manage the overall volume of information they would need to process in order to participate a lot or a little.

The authors of the second paper conclude by saying that they "expect to see more work on large-scale comparative analysis of virtual public discourse dynamics for a variety of technologies (Listserv, IRC, Web-based bulletin board systems, etc.)" (206) but I haven't come across a study yet that specifically compares mailing lists to other platforms.

Mailing list case studies

One case study^[3] of a mailing list centered around a particular open-source software development project concluded that the frequency with which postings to the list went unanswered was a significant source of frustration among participants. You might consider whether this is a problem for your community, and whether it would be more or less of an issue on a web forum based on your familiarity with the user base.

Another mailing list case study^[4] observed that as the community grew, messages broadcast to the list for the sole purpose of disseminating information increased in proportion, compared to messages that were part of interactive discussions. The authors conclude by suggesting that, "if the overall trend toward list usage with low levels of interactivity continues, it may in the long term reduce members' motivation to use [the list] for serious discussions" (322). They also suggest, as potential improvements, splitting the list into sub-lists to address different uses; providing a mechanism for users to filter which categories of message they receive; and augmenting the list with "new and more effective communication media such as social networking tools, wikis, and blogs" (324). If your mailing list community is similar to the one addressed by that case study, you might consider exploring those suggestions before migrating entirely to another platform.

In case of opinion, quote someone else

Finally, though it's not of the same rigor as a journal article, I did come across a blog post^[5] that talks specifically about the pressure for mailing lists to move over to web forums:

I think the discussion has come up at least twice on every mailing list I'm on: Why don't we move this discussion to a web-based forum?

[...]

I got to thinking about it -- why do I, and so many other competent, technical folks prefer the archaic mailing list format to an online forum?

The answer is simple: quantity. I think that those of us who prefer mailing lists simply participate in more different communities.

It would be gratuitous to quote the entire post here, so I'll summarize. The author sees web forums as ideal for welcoming new users and as a space "where participants invest a lot of time on a daily basis." The down-side in his view is that web forums tend to demand this greater investment of time because there are so many different types of forum software to learn, as opposed to mailing lists which all come to the same inbox.

He also points out that that lower barriers to entry result in a lower signal-to-noise ratio, and the public nature of posts attracts much more spam. Dealing with these certainly requires more leadership and moderation activity than some communities are able to provide. (That being said, I've seen these roles transform users into more productive community members, in a way that I think was a net gain for the community.)

He concludes:

...in spite of all the flaws of mailing lists, it's much easier for people to participate in many communities at once when they all come to one place -- email.

You will find lots of differences of opinion, but in the open source world, a huge number of competent developers hang out on mailing lists, and simply do not have time to participate in forums. If you want to reach them, you go to where they are. And lots of other communities grow under similar circumstances.

Wrapping up

My conclusion in turn is that it's part of a community administrator's job to be familiar enough with a community to make an informed decision about which platform is going to ultimately be best for the community. What's "best" should be defined in terms of specific outcomes, and trade-offs have to be identified. Your goal might be to increase the size of your user base, but that could conflict with a goal of retaining current users, especially if there's a difficult transition between platforms or the user base increases too rapidly. If the research shows anything, it's that there's no one right answer.

One thing to add: It's great that this discussion took place openly within your community. There are cases when founders or owners take matters into their own hands and decide what's best for a community on their own. I wouldn't say that never works, but it certainly doesn't hurt to test the waters a bit and get some user feedback about proposed changes.

---

1. Jones, Q., & Rafaeli, S. (2000). Time to split, virtually: 'Discourse architecture' and 'community building' create vibrant virtual publics. Electronic Markets, 10(4), 214-223.

2. Jones, Q., Ravid, G., & Rafaeli, S. (2004). Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Information systems research, 15(2), 194-210.

3. Toral, S. L., Martínez-Torres, M. R., & Barrero, F. (2009). Modelling mailing list behaviour in open source projects: the case of ARM embedded Linux. Journal of Universal Computer Science, 15(3), 648-664.

4. Yu, M. Y., Lang, K. R., & Kumar, N. (2010). Supporting better communication in academic communities of practice: An empirical study of AIS/ISWORLD. Communications of the Association for Information Systems, 26(1), 16.

5. Locke, J. (2010, March 10). Mailing list or forum? A theory... [Blog post].

Answer 2

I don't have data to cite, but I believe I can offer some useful reasoning. My observation on this is that web forums require a higher level of user engagement overall, due to the additional level of difficulty in actually making use of the medium.

Mailing lists, by contrast, are a "don't call us, we'll call you" approach. The users give you their calling card (email address), and you do the rest. Messages go right to their inbox, (the majority of email users have never used a filter^{[citation needed]}) and responding to or posting to the list is a simple function that everyone is familiar with.

For huge swathes of internet users, the idea of even going to a website can be hugely challenging. Google and bookmarks can help, but these are extra steps that must be taken. After you get to the site, then you have to figure out how to use what is essentially a brand new application. Depending on the aptitude of the community's user base, a shift from email to web could kill it, or cause a schism.

In other situations, where a large enough portion of the user base can successfully make the transition, those that do will be more invested in the community simply because of the time and effort required to stay a part of it.

So essentially by this reasoning, the answer to your question depends heavily on the technical proficiency and willingness of the user base in question to invest time in the community.

---

There's certainly room for "the best of both worlds" in web software platforms. Mailing list platforms can be intelligent enough to ingest emails into threaded conversations on a web forum, and provide email digest, RSS, or regular forum thread status icons to keep people informed of the same discussions.