12 Questions T&S Leaders Should Be Asking

Not all platforms are alike. Factors like platform size, organizational structure, and the nature of online threats can influence a platform’s Trust & Safety strategy. Despite these differences, the fundamental objective of Trust & Safety teams remains the same: ensuring a safer online environment. 

In order to achieve progress toward the end goal of ensuring online safety, every aspect of a platform’s Trust & Safety approach should be regularly evaluated. Identifying areas of growth and improvement requires an evaluation of all aspects of Trust & Safety, from team management and impact analysis to policy development and compliance. As part of this process, here are the questions Trust & Safety leaders should be asking.

Team Management 

Team assessment is important for every team leader. This especially holds true for Trust & Safety leaders, whose teams are regularly exposed to high volumes of harsh content that can impact their health and well-being. 

1. How can I keep my team healthy and resilient? 

Content moderators face a high volume of malicious content daily, resulting in high rates of burnout and turnover. While prevention focuses on operational solutions, risk mitigation provides ways to enhance the well-being of Trust & Safety teams. This includes: 

• Mindfulness practices that limit hours of exposure to harmful content
• Access to physical and emotional support sessions with mental health professionals
• Wellness activities, such as exercise and resilience training sessions
• Investment in tools that enhance moderator well-being 

Measuring Content Moderation

Successful leaders should regularly evaluate their team’s efficiency. Enforcement rates, threat coverage, perception of your platform’s work, and fairness, are types of metrics that should be taken into consideration.

2. What is my team’s AHT?

Average Handle Time (AHT) is a core metric of moderator efficiency. It averages the handle times (the time between when a moderator opens a piece of content to the time an action is taken) of an individual moderator, team, or abuse area over time, and measures how quickly items are handled. The more automated actions teams take, the lower their AHT. Although there are a number of measurements to track, AHT is a core metric to monitor.

3. What are my recall and precision rates?

A recall rate measures the percentage of your platform’s malicious content that is picked up by its moderation systems. A high recall rate means that more harmful content is identified, though it can also mean more false positives. On the other hand, precision measures the percentage of items that were identified as violative, which are, in fact, violative.

While most automated detection mechanisms have a high recall rate, and lower precision, solutions that are based on intel-fueled, contextual, adaptive AI maximize both.

4. What tools are available to monitor and improve my team’s performance?

Improving team performance rests on measurement. One way to monitor performance is through dedicated content moderation software that automatically tracks team activities and provides analytics, so you can assess performance and improve over time.

Business Impact

The Trust & Safety teams responsible for overseeing user-generated content (UGC) on various platforms formulate guidelines for permissible content and subsequently apply these policies to implement necessary action. Such policy enforcement measures significantly influence the user experience, engagement, retention, and ultimately revenue – making their work critical to the platform’s bottom line. 

5. What is my team’s ROI?

In today’s cost-conscious economy, the focus on ROI has sharpened and Trust & Safety teams are challenged with doing more with fewer resources, all while trying to scale. For Trust & Safety teams, ROI is incredibly difficult to prove as manual moderation can be both burdensome and expensive. 

After identifying the need for a content moderation solution, teams often struggle between whether to build or buy that solution and can operate under an incorrect assumption that in-house solutions must be cheaper. While it may seem cheaper to use internal resources to build moderation tools, teams quickly find that just like other work tools – moderation platforms are complex. These solutions require specialized knowledge in Trust & Safety, and their development and maintenance may detract the focus of development teams away from the core business. 

Implementing dedicated content moderation software with built-in efficiency features advances operational efficiency by streamlining the moderation process, providing real-time performance analytics and improvements, and implementing custom automation. 

Alignment with Product & R&D

Safety by design is a core Trust & Safety principle, which refers to how technology can proactively minimize online threats from the start. This concept places safety at the forefront of each decision made in a product’s lifecycle and requires regular alignment between Trust & Safety teams, product, and R&D.

6. How do my policy updates align with product and R&D?

While Trust & Safety teams define what can and cannot be posted on a platform, it is up to R&D and product management to ensure those limitations can be executed on the back end, reducing the need for manual detection of policy violations. To ensure that policy updates are smooth and efficient, Trust & Safety leaders should work toward an open, and mutually beneficial relationship with product, where policies are supported on the back end.

Alternatively, implementing SaaS solutions that allow for no-code policy changes can ensure policies are constantly up-to-date, with minimal reliance on external teams.

7. Do I want my product and engineering teams to become experts in Trust & Safety and content moderation?

Before your engineering team can begin building a custom Trust & Safety solution, they will need to grasp complex subjects like machine learning algorithms for content filtering, workflow management, various global regulations, and content policies. To ensure proper system setup, significant upfront research is required. Additionally, ongoing investment is necessary to remain on top of the fast-moving changes across the board. 

Given the reputational and legal risks that can result from failure in your Trust & Safety operation, Trust & Safety experts should be the ones responsible for implementing a comprehensive strategy.

8. Do we have the bandwidth to maintain multiple, custom Trust & Safety integrations long-term?

In order to design the exact content moderation processes desired, an in-house content moderation platform will need to integrate with external detectors/classifiers as well as with messaging apps and case management software, to quickly moderate, enforce policies, manage user flags, send notifications, and more.

Maintaining those integrations vs. integrating once with a single API is another important factor to consider that will impact your team’s bandwidth for core business activities.

9. Which AI tools should I use?

Automated detection is a critical part of content moderation. The AI tools integrated into your Trust & Safety systems should be up-to-date, cover all relevant abuse areas, be adaptive to your decisions, and take into account context. Additionally, using codeless workflows allows for instant policy changes.

Threat Detection

While policies establish the rules of engagement on a UGC platform, policies alone cannot outline risks that Trust & Safety teams are not aware of. Proper threat detection ensures that teams are not left blindsighted in the face of new threats. 

The best way to stop harm is to avoid it in the first place. To do this, teams should be proactively assessing risks, and creating policies to stop them, before they reach platforms. Establishing a trend detection or intelligence team is one way to do this.

10. What abuse areas can my team successfully handle?

UGC platforms face a wide range of abuses, ranging from CSAM and misinformation to hate speech and the promotion of terrorist content. Each of these abuse areas requires specialized knowledge and threat detection activities. Effectively detecting this type of content requires robust, multi-faceted teams. Building an in-house team of experts is one way to do this, but hiring an external threat intelligence team may ultimately offer a more cost-effective way to ensure full coverage of all risks. 

11. Can my team proactively detect geopolitical threats?

Identifying threats before they reach your platform helps proactively detect harm. Geopolitical events, either planned (like elections) or unplanned (like wars and natural disasters) all quickly create new risks, placing huge strains on global T&S teams. Proactive insights into these events are not always possible, but identifying the threats before they manifest on your platform is one way to minimize risks.

Ensuring Regulatory Compliance

As countries around the world create new laws that establish platform liability for harmful content, Trust & Safety teams are required to create policies that are compliant with a diverse set of global laws. 

12. How do I ensure compliance?

As a result of emerging online safety requirements in many parts of the world, Trust & Safety teams must consider how to integrate compliance into their processes. Understanding exactly what is required of regulations in each country where they operate, and what the legal implications are is critical to ensuring compliance.

Trust & Safety solutions with built-in compliance features not only help teams get a grasp on what they need to do to be compliant, but these features also help them achieve compliance. For example, for compliance with the EU’s Digital Services Act, ActiveOS’s out-of-the-box solutions support platform transparency, user flagging, appeals, and notices processes, to name a few. 

To increase the efficiency of your Trust & Safety team, ActiveFence offers a number of tailored solutions for platforms of all sizes, including:  

• ActiveOS provides a free and fully customizable content moderation platform with no coding required, allowing Trust & Safety teams to manage their entire operation in one centralized place. 
• ActiveScore helps Trust & Safety teams make faster decisions with greater accuracy through automated AI content detection fueled by insights from more than 150 intelligence researchers. 
• ActiveFence Threat Intelligence solution offers proactive off-platform intelligence collection and analysis in more than 100 languages, across a variety of abuse areas to effectively counter emerging threats.

To better understand how ActiveFence can help streamline your work, request a demo.