Reddit Study Shows Power of Community Guidelines in Combating Harassment

Trolling and harassment are so common online that they may seem unavoidable. But what if things didn’t have to be this way? New research on one particular corner of Reddit suggests that online communities are actually a lot more responsive to ground rules of civility than one might expect.

In a new community-based experiment, researchers found that pinning a simple reminder of forum guidelines to the top of posts in the popular web forum r/science had a significant impact on increasing new users’ contributions to discussions and on decreasing the amount of abusive or off-topic comments in the threads.

In a paper published Monday in the journal Proceedings of the National Academy of Sciences, Nathan Matias, Ph.D., shows evidence that a “stickied” post at the top of a comment thread was associated with an 8.4-percent higher likelihood that new redditors’ comments wouldn’t be deleted for violating community guidelines. He also observed that posting the rules correlated with a 70-percent higher chance that first-time commenters would jump into a discussion. Matias, an associate research scholar at Princeton University, writes that the reminder may help new commenters feel more comfortable joining the conversation by making it clear that the group abides by some social norms.

“r/science is one of the largest discussions of scientific topics in the world with over 20 million subscribers,” Matias tells Inverse. “The prominence and scale of these conversations makes them ripe for conflict, harassment, and other forms of harmful comments, especially on controversial issues.”

But the results of this study provide early evidence that even in a massive web forum, simple suggestions can have a significant effect.

“By providing information about how other people will behave, they also have a strong effect on a newcomer’s decision to participate in a community for the first time,” he writes in the paper. In short, the reminder helped newcomers understand what kind of behavior is and isn’t tolerated, not just from them, but from everyone else, too.

Matias gathered the data for this study over a one-month period in 2016, when r/science had about 13 million subscribers (that number is now over 21 million). To conduct this experiment, Matias’ CivilServant software, which the moderators of r/science granted moderator privileges on the subreddit, automatically posted the community guideline reminders in some discussion threads and then observed how commenters’ behavior in those threads compared to those without the reminders.

“In our early conversations, people in the r/science community expected very different outcomes,” Matias says. “That’s the great part of citizen science experiments, which allow communities to test rival hypotheses and discover the answers for themselves.”

Piper Below, Ph.D., an assistant professor of medicine at the Vanderbilt University Medical Center, was one of the people in the community who helped test these hypotheses. In her role as an r/science moderator, she helped manage the relationship between Matias and the rest of the subreddit. Below tells Inverse that when Matias proposed this project, it seemed like a natural fit.

“Being a moderator of r/science, all of us are scientists in some capacity or another, and so when somebody approaches us interested in studying r/science, it’s sort of a natural thing for us to want to entertain the idea of that kind of collaboration,” she says.

But it was more than just scientific curiosity that led the moderators to cooperate with this experiment.

Since r/science is the subreddit with the sixth-highest number of subscribers on Reddit — out of thousands of subreddits on the entire site — enhancing the quality of discussions is the best way to ensure that the primary purpose of the forum can remain intact.

As Below explains, the more all users follow community guidelines, the better the user experience is for everyone. And by the same token, the more poorly people adhere to the rules, the less useful the forum becomes.

This is especially important for newer users. One of the most frustrating experiences for users, says Below, is to come to a post they’re excited to learn more about,only to see a so-called “comment graveyard” — a vast thread filled with hundreds or thousands of comments that the moderators have deleted for failing to follow the community guidelines. She hopes this study will help her and her fellow moderators develop new ways to foster constructive discussions.

“If we can find any approaches to website interface or content layout or design, rule prominence, et cetera, that would help improve the Reddit user compliance with our rules. Then, that results in fewer deleted comments and a commenting environment that feels more engaging to the reader,” she says. “That’s something that we take very seriously at r/science, so whatever we can do to cultivate a space where the commenters feel like they understand the rules and know how to comment within those guidelines is only good for creating a really positive, active forum for science communication.”

Matias emphasizes that the newcomer’s experience is vital to the health of the subreddit since the wide variety of levels of scientific understanding is the forum’s strength.

“Experienced scientists have an opportunity to discuss their research and discover how people see it. Students have a chance to go deeper into the questions they’re studying in class,” he says. “And all of us have a chance to participate in a conversation about new discoveries where we may not be experts.”

He’s also careful to point out that conflict is complex, involving societal, individual, and cultural issues, so no one solution will be a silver bullet in all cases.

“I have shown how social norms can reduce the rate of harassment in r/science, but it’s important to put this finding in context,” Matias says.

As in any good science, Below notes, the next steps will be to do follow-up analyses and additional replications — which they’re currently in the process of designing in collaboration with Matias.

One feature she says they’re considering implementing will actually accommodate one of the most common violations of community guidelines rather than attempting to prevent it.

Rule number 8 of r/science says, “No anecdotal comments,” which means that if the post is about a peer-reviewed scientific study on a particular topic, users are not supposed to chime in with their personal experience — no matter whether it confirms or refutes the findings. This rule is meant to keep the discussion focused on the research at hand.

But Below acknowledges that when people break this rule, they are most likely just eager to engage with the discussion, so the moderators are considering a feature that directs users to post these anecdotes under a pinned comment, which will be automatically collapsed when users navigate to the page. Therefore, users who want to engage with it can click to expand it, while everyone else can safely breeze past it without the conversation being derailed.

No matter what the moderators’ future interventions look like, though, Below says it’s all in the name of making r/science a better place for people to learn about new peer-reviewed research and discuss it with each other.

“Most communities on Reddit are striving to give their users the best experience within the scope of the goals of their subreddit, and I think that work like Nate’s is exciting and essential for really quantifying user and moderator behavior and how we can really get to the best user experience.”

Abstract: Theories of human behavior suggest that people’s decisions to join a group and their subsequent behavior are influenced by perceptions of what is socially normative. In online discussions, where unruly, harassing behavior is common, displaying community rules could reduce concerns about harassment that prevent people from joining while also influencing the behavior of those who do participate. An experiment tested these theories by randomizing announcements of community rules to large-scale online conversations in a science-discussion community with 13 million subscribers. Compared with discussions with no mention of community expectations, displaying the rules increased newcomer rule compliance by >8 percentage points and increased the participation rate of newcomers in discussions by 70% on average. Making community norms visible prevented unruly and harassing conversations by influencing how people behaved within the conversation and also by influencing who chose to join.

Editor’s Note: As of 4:38 p.m. Eastern, 4/29/2019, this story has been updated to include original comments from Nathan Matias.