Policy Series: Illegal, Violent Activities

Background

Across the internet, violent and criminal content is being created, uploaded, and shared by dangerous individuals, organizations, and communities. Ranging from content that glorifies criminal and extremist organizations to promotional materials facilitating the trade of illegal goods and services. Graphic violent imagery is shared not only by terrorist groups looking to shock civilian populations but also by radical far-right groups seeking to bully and harass minority communities online.

To keep users and platforms safe from this content, companies must create robust community guidelines and policies that are constantly updated to accommodate ever-changing on-platform behaviors and shifting legal requirements.

The following article and accompanying report are the second in ActiveFence’s Policy Series that analyzes the policies and community guidelines of some of the biggest online platforms to help combat illegal and violent activities. The policy areas in this report can be divided into three categories: dangerous individuals and organizations, graphic violence, and hate speech and incitement.

ActiveFence has reviewed twenty-six of the leading technology platforms that host user-generated content in sectors ranging from social media, video and file sharing, video conferencing, and instant messaging. We have synthesized the policy into a simple, easy-to-use summary of the current approaches to tackling platform abuse.

Policy Challenges

The speed and scale of change in the online world makes policy creation a complicated endeavor. As a result, user content policies must be rigorous and detailed, but they should also be non-exhaustive. The challenge faced by policy builders is building a system for evaluating content that protects the spirit of the wording and can respond to new and evolving threats. In addition, companies must understand their place within a dynamic digital environment—both as it is today and as it will be in the future.

While this may seem like a daunting task, ActiveFence is here to help. We will report on evolving community guidelines and policies and will regularly update as the industry changes. This report is not a suggestion for writing policy itself but is an essential tool in making informed decisions and checking marketplace synchronization.

A Platform Policy Guide

This guide provides valuable insights into how various platforms—and types of platforms—work to keep bad actors from abusing their services and users. Complete with examples and divided by platform category, this report will feature an analysis of three abuse areas: dangerous individuals and organizations, graphic violence, and hate speech and incitement.

For each risk area, we will provide the responses of the five different types of digital platforms: social media, instant messaging, video conferencing, video sharing, and file sharing. This is because platform type comes with its own unique risk areas that dictate the necessities and requirements surrounding policies related to these issues.

Social Media

Social media platforms enable almost all forms of user-generated content to be shared, from audio, images, video, and text (both in instant messaging and posts). They also enable the sharing of third-party links and the creation of groups, both private and public.

Due to their diversity of use, social media platforms are open to perhaps the greatest volume of risk of abuse by dangerous organizations seeking to coordinate their supporters online. This also places users of social media platforms in danger of exposure to highly graphic content and leaves them vulnerable to hateful and inciting content.

Instant Messaging

Instant messaging platforms are abused by dangerous organizations coordinating illicit group activity, such as the sale of illegal goods or the pursuance of criminal activity. Furthermore, individuals seeking to shock or harass others frequently share graphic gore content on instant messaging platforms as part of the abuse. Lastly, hate speech and incitement are also issues of growing prevalence on instant messaging platforms and applications.

Video Conferencing

The COVID-19 pandemic saw an exceptional growth in the use of platforms providing video conferencing services. Unfortunately, this popularity also made these platforms more susceptible to abuse, including by terrorist and criminal organizations and violent extremists. For example, digital raids by hateful extremists often include bombarding participants with graphic, violent, and hateful content.

Video Sharing

Video sharing websites provide a platform for people to interact, share, and discuss various topics. Unfortunately, certain individuals and organizations also seek to abuse these platforms to spread hate and incite violence. Indeed, the ability to easily upload video content enables bad actors to spread harmful material, utilizing extreme materials to capture global attention.

In particular, video sharing platforms have come under specific abuse by terrorists and other violent extremist groups who abuse the platforms to spread video propaganda or upload links to their off-platform content. Organized crime also targets the platforms to promote illegal goods for sale.

File Sharing

Bad actors, including terrorist organizations and hate groups, have abused file sharing platforms to promote terrorist activity, racial hatred, and violence. For example, white supremacist groups and individuals are known to abuse file sharing platforms for the purpose of distributing materials, manifestos, and music that encourage violence against other ethnic groups. The result has been real-world carnage, including mass shootings committed by white supremacists in Christchurch, New Zealand, Charleston, South Carolina, and elsewhere.

Shaping Responsive Policies

Nevertheless, the best platform policies are responsive and should evolve as new threats arise and change over time.

For details on how some of the largest online platforms currently address the threat of illegal and violent activities on their platforms, download our comprehensive report.