How do we maintain answer quality when everybody thinks they're experts?

Question

I participate on several Stack Exchange sites that have scope that's a little more subjective than the norm. These topics tend to also be more "accessible" to people; you pretty much have to be a programmer to answer questions on Stack Overflow, but anybody who once knew somebody who had a cat might feel qualified to answer questions on Pets. Ditto topics like the workplace, health, writing, parenting, and lots of others -- lots of people have some experience with the topic (or maybe just some speculative theories), but answers work best when they come from a position of relevant expertise. But, to add a wrinkle, sometimes expertise does come from experience; it's just that not all experience leads to expertise.

We would like to raise our overall answer quality, which I think means a combination of attracting more users who can contribute high-quality answers and discouraging the less-informed answers. In this question I want to focus on the latter.

We'd prefer to help the users who are posting the speculative, anecdotal, answers to do better; I'd rather they improve their contributions than go away. People already leave constructive comments and downvote (not enough of the latter, but some) on individual answers; I'm looking for things we can do at a broader level. Ideally I'd like to intercept these answers before they are posted -- that is, find a way to help users think about what's really needed and then do that.

What have other communities done to address this? I know of one Stack Exchange site that has a flowchart for questions, but I haven't seen anything like that for answers. Some sites have rules about citing sources, but that isn't a complete solution -- sometimes an experience-based answer is appropriate. (I imagine that DIY would be very different if that weren't permitted.) What has worked for others?

While this question arises out of Stack Exchange, this isn't a technical question. I'm really looking for a behavioral solution, and there are other communities besides SE that may have addressed this issue.

Answer 1

I'm part of a Stack Exchange site that had very similar problems (Mythology). During the private beta, we had a lot of answers that were incorrect, or that didn't go into enough detail. It was clear that many people were not doing any research, and as a result did not have in-depth knowledge of the subject.

Here are the strategies that we used to raise the quality our questions and answers:

Ask expert questions that require expert answers

The quality of an answer is partially reflected on the quality of its question. If a question is basic enough that it can be answered anecdotally, then it will be answered anecdotally. Ask questions that are so difficult to answer that they require people to do research. Ask questions about unusual situations, so people can't rely on their memory when they answer it ("that happened to me several times" v. "that never happened to me").

Lead by example

Is anyone writing non-anecdotal answers on that site? If no, then someone should start doing so. You can't ask people to write non-anecdotal answers if there are no examples of non-anecdotal answers on your site.

Require (or at least heavily encourage) reputable sources

Requiring sources is practically guaranteed to raise answer quality for a number of reasons:

1. Answers with sources are much less likely to be incorrect. If you can remember the source of your information, you probably also remember the information correctly.

2. Answers with sources are much more likely to be detailed. Again, if you can remember the source of your information, you probably remember minor related detailed that are relevant to the answer.

3. Answers with sources have other benefits as well: witness this exchange on the History meta site:

   Is this site an academic source?

   My teacher recently rejected the idea of me using this site for a research paper, and I was wondering if you would think that this site could be considered a source.

   ---

   Your teacher was right to do so. This website would probably be a bad source to use as a primary reference in a paper. Any random moron in the world with internet access is free to post an answer here. (For exhibit A, click my name below...)

   However, if there's something that has you stumped in your research, I'd think it would be an excellent place to try to get your questions answered. Good answers here should be sourced with hyperlinks, so it would also (hopefully) be useful for digging up other references that are usable. Use the references we supply to help jump-start your research.

   Answers without sources are going to be useless to experts (specifically experts in academic disciplines), because they won't be helpful when doing academic research.

The risk with requiring sources is that you disallow the kinds of answers written by people who have lots of experience in a topic but who didn't learn about the topic in an academic setting, and thus don't have any written sources for their knowledge (e.g. textbooks, papers, etc). I have two things to say about this:

1. A "source" could be a picture, a section of code, or a video. To use the example of the DIY Stack Exchange, part of writing a well-sourced answer could be taking a video of a technique, and using that video to back up the answer.

2. You don't need sources in real life, because you know how much you can trust a person giving you information. For example, if Professor X from Harvard University says y, you can probably assume that y is true. Unfortunately, this approach doesn't work on the internet. Someone who says "do y: you can trust me because I have ten years of experience and I'm an influential person in the subject" could very easily be lying.

   One solution would be to verified users' identities, and highlight the profiles of "experts" in some way. Unfortunately, Stack Exchange is philosophically opposed to this. This leaves sources as the only way to verify that an answer is accurate.

Experts will not participate in a non-expert community

To address the first part of your question ("attracting more users who can contribute high-quality answers")...

You will not be able to attract experts if you have beginner level content. Think about it: if you are an expert in a field, you want to learn from other experts ¹. That means participating in communities where there are other experts. If you don't have any experts (low-quality answers is an acute symptom of having no experts) in your community, then you won't attract any experts. What you need to do is raise the quality of your answers, teach your contributors how to do research (so they can learn and become experts), and then promote the community to other experts.

Quantity over Quality

You write:

We'd prefer to help the users who are posting the speculative, anecdotal, answers to do better; I'd rather they improve their contributions than go away. People already leave constructive comments and downvote (not enough of the latter, but some) on individual answers; I'm looking for things we can do at a broader level. Ideally I'd like to intercept these answers before they are posted -- that is, find a way to help users think about what's really needed and then do that.

That's unrealistic for two reasons:

1. You aren't going to reach everyone. Some people are incapable of writing good answers, no matter what you do. You have to focus on reaching the people who can.

2. You might not be able to reach users before they post answers. Most times people learn best by posting an answer, receiving comments and downvotes, and trying again.

You are probably going to loose people (and activity) if you make your site more expert oriented. You need to ask the community: is this a trade-off we want to make?

---

¹ No, most experts do not want to participate in communities just for Fake Internet Points™ or the joy of hoping beginners. Experts have limited time available: in most cases, the only way they can justify spending time on a website is if it helps them (whether by advancing their career, by teaching them about things they didn't know, or by helping them become a better expert).

Answer 2

I guess you are well-aware that we on such a SE site. Community Building is a subjective topic as DIY or Physical Fitness are.

You must make clear that not every answer is suitable for every question. This doesn't start with the written answers of your non-experts, it starts with the questions written by uninformed users. Users ask questions to receive answers to solve their problem as fast / good / efficient as possible. But the answers they get are primarily based on what they ask. Somebody asking for personal experiences gets different answers compared to someone asking for scientific answers.

Hey guys,
I need some help from you. Can you explain to me the difference of this and that, please?

There you go. The user who would have written this question would receive a broad load of answers. Some will be backed up with personal experiences, some will be backed up with scientific clues of persons who really know their subject.

Hey guys,
I need some help from you. Can you explain to me the difference of this and that, please? I need some scientific evidence for my research paper.

Nobody with personal experiences answers this one. These users are deterred by the asking of scientific evidence. Anecdotes and stories from your neighbours aren't scientific - no doubt about that.

Hey guys,
I need some help from you. Can you explain to me the difference of this and that, please? I really need some personal advices in this matter.

This is the moment for your experience-based users, they will go crazy to answer questions like this. Everybody will try to answer the question by sharing their personal experience gained in years of doing this and that.

In the end, you can solve a lot by telling your asking users to follow some crucial rules. Add a quick advice in the sidebar of your site (if it's possible) which politely reminds of the correct form of asking questions. Make clear that differently written questions get different answers.

Summary: On the average Web page, users have time to read at most 28% of the words during an average visit; 20% is more likely.

Well, there it is. The inability of some users to read. You always come across users that clearly don't want to read your well-elaborated rules for asking questions. They don't want to hear all your bureaucracy stuff, they want answers for their questions. You can't really do anything against such behaviour. Ask them politely to read the rules in order to get the most satisfying answers. Tell them that your community really wants to help them but they need to cooperate.

However, there is always the vice versa case, too. A well-written question demanding scientific evidence or personal experience receives a lot of inappropriate answers. A strategy to prevent those answers:

• Elaborate on your meta site which answer type should be given when. Don't do that alone or even with your team, it's really important to let the community participate in this event so they can design and understand the rules. Don't do any unnecessary changes to those rules. Every change will lower the acceptance of the rules since users see that they could be changed and influenced by them.
• Remind to those rules as often as possible. If you see a good answer, write a comment linking to your written rules and say "This is an answer really fitting into our set of rules" or something similar. If you see an inappropriate answer, write a comment linking to your rules again and tell them to follow the user designed rules. It's important to make those rules an integral part of your AQ culture.
• Discuss inaccurate answers. You may come across an answer having both, experiences and evidence, still both isn't well-elaborated. Don't vote it down or up, don't delete it. Discuss it. Make sure that such answers don't show up anymore. You don't want to discuss again and again, so link back to your previous discussion and the rules for every answer you get.

It's important to understand that you can just increase the possibility of getting highly elaborated answers by enforcing rules. There are users who can post good answers although there aren't rules around, but this part of your community (usually) is little.

Concluded you should try the following:

• Try to set up rules for questions and answers to make sure what is needed when.
• Remind to them as often as possible.
• Use up- and downvotes to rate questions / answers. These vote should be mostly based on your rules.
• Encourage your users to write comments to remind to the rules. Your users should ask for clarification if they are not sure what actually is needed.
• Delete answers which aren't edited in a particular time span to show how important good answers are for you. (The time span should be fair, at least 14 days or more so users can react to such requests in time.)