Counterintuitive: Most organic communities (typically of a small enough size) are self-sufficient like this, because if they are not, they have likely already broken apart and died. Hear me out.
When people congregate in a certain place, for any purpose, they do so because they share a set of communal values. These values are varied and implicitly derived from community members. For example, values span from "we enjoy this game" to "it's nice to come home and relax with online friends" to "it's okay to shit on gay people." I pick this last example specifically to highlight that they don't have to be good values, they just have to be shared precepts that people are willing to invest in.
When an organic community forms, a few things happen:
- The people integral to the community become - formally or socially - recognized as such.
- The integral people set standards and values - either transcribed or socially inferred - that others in the community are expected to follow or agree with.
- Dissidents are dropped from the community (via any number of processes), reinforcing integrality.
So, when do these communities self-moderate? Whenever the feedback loop driving integrality successfully reinforces itself. Why? People who don't adhere to the communal values are ejected, socially or literally, or otherwise leave. That act is self-moderation. The end-state is a community where (functionally) all members agree with the core values, and enforce them, even if the primary leaders are not immediately available.
For example: I used to play OSRS casually. In 2017, the pride riots happened, and "it's okay to tell LGBT people to piss off" was adopted as a communal value. I'm queer on many axes. I was never forced out, but I felt socially isolated, and left. That was self-moderation. It was bad self-moderation; it was poorly aligned with our values; it was still self-moderation.
Another example: a community I am in has a precept, that we do our best to value all people. One person in specific kept talking about how they'd wish billionaires would die. Many people in the group gave them enough challenge and negative feedback that they eventually felt pressured to leave - their values no longer aligned with ours. This was unpleasant, and broke long-standing friendships, but it was in fact self-moderation in action.
A third example: suspensions on Stack Exchange inform someone that their values for their conduct are out of line with ours, and that they now need to weigh their desire to participate against their desire to act that way. More people actually do this evaluation than you'd think.
A fourth and final example: My workplace has a value, "deriving profit for the corporation is an admirable personal goal." I disagree with this value for philosophical reasons. But I don't voice that as disagreement - instead, I compromise to it, because other factors, such as access to interesting problems and a strong community, is a trade-off I can accept. In other words, compliance is an alternative to outright agreement.
(One important note: I'm taking a step back from evaluating the values of a community. I think this is important to do. Our colloquial usage of "well moderated/unmoderated" often carries with it a value judgement about how a good community should be behaving. In reality, diverse communities will behave in different ways, and applying our values judgement on the self-moderation side is misplaced.)
When you ask about self-sufficiency, this is what it looks like: a community has values, and maintains them via enforced agreement. When these values aren't externally proscribed, they're implicitly derived from the Important People within the community and are no less existent.
Now on to your comparative questions. I am going to take the liberty of rewording them somewhat, to fit with this mental model.
- Is a self-sufficient community with only social moderation is the optimal model whenever it actually works?
"Optimal" is tricky to define. This kind of community is effective, because at any given moment, the members of the community are highly likely to be compliant with its values - otherwise they wouldn't be there. It's not much fun participating with people you can't stand.
But they do have costs. First, their goals can never be as broad as a community with a more solid set of guidelines. Stack Exchange is a good example: if we used social guidance only, it would be a disaster, because Stack Exchange is an information-first and not a community-first website. In the absence of codification, we would see far more toxic contributors who are here for the information and not the community. The Linux Foundation is another good example of why codification of a code of conduct - and recalcitrance towards it - encourages toxicity to minority members.
Second, values questions can emerge, and these can be fractious when people fight to inscribe their social values over others'. As an example: two family members arguing about abortion at a gathering can polarize people into pro- and anti-choice factions. Eventually, either the community will split along those lines, or they will reach a tenuous compliance, where whether one cares about abortion rights is deemed less important than maintaining familial ties.
But the stability you see typically comes after these major challenges have shaken out. If you look at any successful long-standing community without codified standards of behavior, I can almost guarantee its past was tumultuous, with comings and goings, fractional sub-communities, et cetera. By the time you're most likely to see it, the community values will be set - the only question is whether they now fit you.
- Under what conditions can an online community (with free registration) become self-sufficient in that way?
Most communities get there eventually, I think - if they don't evaporate before they do. Evaporation is a real risk when fractious issues become emotionally exhausting, and group participation turns from rewarding to taxing.
But if a community makes it through that... they're home free.
- If it seems the community can become self-sufficient, is it optimal if the moderators simply let it go and refrain from stepping in, unless a spammer appears?
Optimal when used here implies a value judgement. It is generally more expedient in the handling of dissident viewpoints if a moderator can step in and handle the question through manual enforcement. But it makes the remaining community less solid to do so - now, instead of agreement, it is instruction.
Sometimes that transition is okay, when it's safe to assume moderators are only enforcing outstanding values. For example, I have once banned a pedophile from a friend group server. I highly doubt anyone would have challenged me on that decision if they understood my rationale. (There was a values discussion about transparency.)
And sometimes I think it's better to leave a question up to the community, when the rules haven't yet been codified or aren't so clear-cut. It takes a keen eye to tell when this is the case, though, because by nature these cases will feel clear-cut to the moderator/community leader. For example: "he was coming onto her in PMs in an overtly inappropriate way, but it was off-server. should we suspend him?" If you ask anyone this question, you will get two vehement responses: a) yes, because harassment is an inappropriate activity for anyone who wants to participate here; b) no, because it happened elsewhere, and doesn't fall into our domain of moderation.
...and sometimes a statement is both fractious, and warrants obvious and immediate enforcement. For example, "posting images of underage girls is okay, if and only if they're drawn" is, believe it or not, a fractious statement in many communities, no matter how obviously absurd it might sound. Without codified moderation these communities may decide to agree.
- (Because I'm pretty sure that not bothering to ban this spammer was far from optimal, wasn't it?)
Who cares? The community has a value: "we can put up with the occasional spammer." Is that a wrong or non-optimal value? Not really. Not from any objective perspective.
So, you be the judge.