The Trust & Safety Glossary

Refers to the degree to which automated or manual moderation tools make the correct decisions. Sometimes measured as an inverse of the false-positive rate. Read more. 

General term for unwanted on-platform behavior

A reference to the volume of violative content and the flexibility of the system with the capacity to properly moderate the content.

The act of bypassing platform moderation actions or circumventing a platform ban, often involves the creation of at least one additional account.

By creating new accounts, threat actors can return to a platform whose policies they have violated, in an effort to continue to do harm.

Permanently removing or blocking a user from the platform.

Read more: Policy Enforcement

Popular term among predator communities to denote “child pornography”

In Trust & Safety, refers to online risks to children, including exposure to harmful content, abuse, harassment, CSAM, and exploitation.

Read more: Policy Series: Child Safety

Widely referred to as CSAM.

Images, text, or videos depicting the sexual abuse of minors (under 18 years old). For many platforms, this includes individuals who appear to be minors.

Websites that are publicly accessible to all audiences, through standard browsers and search engines. Estimated to only host 5% of online content.

The process of detecting, flagging, removing, or denying abusive users the ability to post harmful comments.

Read more: Content Detection Tools

Moderation activities that are conducted by community members, and not professional moderation teams.

The internal process of screening user-generated content posted to online platforms, in order to determine whether or not it violates policy, and take appropriate action against violative content.

Team member involved in the dynamic process of creating and maintaining the community guidelines of an online platform.

The use of online platforms to bully individuals, generally refers to the abuse of children.

The part of the online world that is only accessible via dedicated dark web browsers which use “onion routing” technology to allow completely anonymous browsing. While the dark web isn’t necessarily illegal or illicit, the anonymity it provides allows for these activities to take place.

A type of content moderation where no individual person makes the moderation decision, rather community members (individual users) vote to determine if an item should or should not be allowed on the platform or forum.

The broad range of actions taken by Trust & Safety teams when content violates policy.

Read more: Policy Enforcement

The rate at which items or moderation events are incorrectly identified. This is the inverse of accuracy.

Read more: Measuring Trust & Safety

Defined as criminal deception intended to result in financial or personal gain.

In Trust & Safety, this may also refer to deceiving a user into providing their personally identifiable information (PII) or unknowingly providing access to their devices or accounts.

Any text, image, audio, video, or other content posted online that is considered violative, malicious, deceptive, illegal, offensive, or slanderous.

In Trust & Safety, organizations such as NCMEC aggregate databases of image hashes that are related to various offenses (in this case, child safety). Platforms can then compare image hashes from their content to hashes of known malicious content (such as CSAM). This way, moderators do not have to view and analyze a potentially harmful piece of content and can compare its hash to that of recognized images.

Read more: Content Detection Tools

Technology that creates a unique, fixed-length string of letters and numbers to represent a piece of data (often an image). The hash is non-reversible, meaning an image can’t be recreated from its hash.

Read more: Content Detection Tools

Any expression or online content that incites, discriminates, justifies hatred, or promotes violence against an individual or group. Given distinct cultural and linguistic nuances, the detection of hate speech is often a complex task requiring regional and linguistic expertise.

The collection of intelligence by means of interpersonal contact.

In Trust & Safety, specialized teams use human intelligence to infiltrate threat actor communities and proactively identify their means and methods.

Apps or websites that are intentionally created to resemble existing apps or services or appear to be a part of the user interface in order to gain access to personal data, passwords, bank accounts, etc.

The creation of fake accounts, often using a target’s name or photo, in order to cause harm to that individual.

In trust & safety, intelligence is used to proactively alert platforms about impending risks, and to inform better moderation decisions.

Internal or vendor teams that are responsible for on- and off-platform intelligence collecting in support of the Trust & Safety team’s efforts. Utilizing a broad range of tactics, including OSINT, WEBINT, and HUMINT, the team detects new threats and trends, identifies tactics, techniques, and procedures (TTPs), conducts investigations into suspicious account behaviors, “red teams” platform policies, and more.

Read more: Trust & Safety Intelligence

A component of a financial institution’s anti-money laundering policy, Know Your Customer (KYC) is a requirement for financial institutions and certain financial technology firms to verify the identity of a client to prevent illegal access to funds (for example – funding of terrorist activities).

Child predator slang for an underage female with a childlike appearance, or a woman who is of age but physically looks, or dresses like a minor.

Content that is created or shared with malicious intent. Includes but is not limited to child sexual abuse material, nudity, profanity, sexual content, bullying, terrorist content, violence, and disinformation.

The unintentional creation or sharing of inaccurate or misleading information. This differs from disinformation in that misinformation is unintentional, while disinformation is the intentional distribution of misleading information.

Platforms can allow users to mute other users so that their activities do not appear on their feeds.

Read more: Policy Enforcement

Sexually-explicit images that were either acquired unknowingly or unlawfully or were taken consensually but shared or posted online without consent.
NCII also include the sharing of intimate imagery beyond the scope of its intended use (ie. leaking intimate images shared on one platform, across other platforms)

Sexual harassment or misconduct that is conducted online. This is disproportionately aimed at women and/or members of the LGBTQIA+ community.

Also knowns as web intelligence (WEBINT). Intelligence that is collected via publicly available information online.
In Trust & Safety, open-source intelligence is used to gain a contextual understanding of harmful activities, enabling proactive content moderation.

The content policy of a user-generated content website defines what can and can’t be posted to that specific website. Also known as Content Policy or Community Guidelines.

Member of the policy team, responsible for analyzing and examining the effectiveness of a company’s content policies.

The team responsible for enforcing the platform’s content policies and taking action against violative content.

Individuals in the Trust & Safety team responsible for establishing the platform’s community guidelines and defining what is and isn’t allowed to be shared on the platform.

Involves collaboration with internal teams as well as external agencies such as law enforcement, regulators, and industry partners.

A form of content moderation that aims to detect malicious or violative content before it is seen or reported by others. Utilizes various techniques, including automated detection and intelligence collecting to identify the violative content before it has a chance to harm user safety or platform integrity.

Previously used only to detect illegal content like CSAM and terror, pending legislation may require platforms to proactively moderate harmful, not just illegal content.

See Harmful Content Detection

A metric that indicates the rate at which action was taken on content or accounts prior to being posted or reported by other users.

Calculated by the number of proactively moderated items divided by the total number of moderated items.

A moderation process that relies on a platform’s community or other individuals to identify and flag content that may be in breach of a platform’s policies. Due to its reliance on the community, violative content is often seen by multiple users, before action is taken.

The measure of how much of a platform’s malicious content is picked up by its moderation systems.

Calculated by the number of correctly identified malicious items, divided by the total number of malicious items on the platform. For example, if a platform has ten malicious pieces of content, and AI identified seven of them, that AI had a 70% recall rate. For most automated detection mechanisms, recall and precision are inversely correlated.

Read More: Measuring Trust & Safety

A method or process where attempts are made to replicate a system, process, machine, device, or software. Used in cybersecurity and red teams to find malicious code in software, websites, and apps.

An enforcement mechanism that involves the temporary, time-limited banning of an account.

Read more: Policy Enforcement

Generally provided by intelligence providers or internal intelligence desks, TTPs are a description of the techniques used by bad actors to conduct harm. In Trust & Safety, an understanding of bad actor TTPs can provide teams with the insights needed to proactively detect and stop on-platform harm.

Learn more: Trust & Safety Intelligence

Prevalent in eating disorder communities, the term references images or other content that encourages one to engage in extreme dieting behaviors. Also known as “thinspiration”, “bonespo”, “fitspo”.

Learn more: Eating Disorder Communities

Content that is correctly flagged as violative

Teams that are focused on the development, management, and enforcement of a platform’s policies to ensure that only acceptable content is posted online and that users are safe.

An individual or vendor who is considered by the platform to be an expert in their field. Content flagged by this entity is therefore given special notice by moderation teams.

Read more: Content Detection Tools