How has the work of Trust & Safety evolved over the past decades, and what lessons from its history can guide today’s complex online safety challenges?

Metadata

Summary

In this preprint chapter from the forthcoming volume Trust, Safety, and the Internet We Share: Multistakeholder Insights, authors Toby Shulruff, Jeff Lazarus, and Amanda Menking—researchers and practitioners with deep expertise in online safety—present findings from a qualitative, interview-based study mapping the evolution of the Trust & Safety field.

Through interviews with 63 professionals across continents, sectors, and roles, the authors trace the field’s history across three distinct “waves,” identify the technological and societal forces shaping each era, and unpack the promises and pitfalls of professionalization, systematization, and mitigation strategies.

The paper covers the following:

  • How early online communities balanced openness with intervention, and why “never hands-off” was the reality from the start
  • The explosion of global connectivity in the mid-2000s and its impacts on online harm, activism, and moderation practices
  • How recent events—livestreaming tragedies, political polarization, and generative AI—reshaped Trust & Safety work
  • The rise of professionalization, from informal networks to industry associations, and the risks of narrowing diversity
  • The systematization of policies, tooling, and outsourcing, and the labor concerns they raise
  • The growing emphasis on proactive and preventative approaches like Safety by Design
  • Regional inequities in resources, tooling, and access for CSOs outside the Global North
  • And more!

Takeaways

1. Trust & Safety’s three historical waves reveal enduring challenges and shifting contexts

The authors’ interviewees traced the field from the Early wave (1990–2004) of ad hoc moderation and volunteer-led community management, through the Mid wave (2005–2017) of massive scale, diversified harms, and global expansion, to the Recent wave (2018–2024) marked by political pressures, AI-fueled threats, and heightened public awareness.

  • Early wave: Online communities were moderated manually, often by volunteers, with inconsistent policies and little cross-platform coordination.
  • Mid wave: Social media, smartphones, and sharing economy apps accelerated both connection and harm; professional teams, advisory councils, and cross-platform collaborations emerged.
  • Recent wave: Trust & Safety became a recognized profession, but also faced intensified abuse tactics, livestreaming crises, and resource inequities.

This framework underscores that many “new” issues—harassment, scams, CSAM—are persistent, evolving alongside technology.

2. Professionalization has transformed the field, but it comes with tradeoffs

Interviewees described a shift from word-of-mouth hiring to specialized, subject- and language-specific roles, supported by formal associations like the Trust & Safety Professional Association (TSPA).

  • Benefits: Greater recognition, information-sharing, common practices, and strategic influence within companies.
  • Risks: Loss of diversity if credentials outweigh lived experience, erosion of volunteer contributions, and potential over-academization.
  • Mixed impact: Smaller, newer companies often integrate safety earlier, while larger or older firms may deprioritize it during layoffs.

Professionalization’s success hinges on retaining diverse pathways into the field while formalizing standards.

3. Systematization and automation have increased scale and consistency while raising equity and labor concerns

Trust & Safety teams have adopted detailed policies, automation for detection and takedown, and outsourced operations to BPOs, vendors, and consultants.

  • Advantages: Faster, more consistent decisions; scalability for smaller companies; bulk removal of harmful content.
  • Challenges: Outdated or biased AI models, inadequate support for underserved languages and regions, erosion of contextual judgment, and poor labor conditions for outsourced workers.
  • Global gap: Many tools perform well in English but “fall off a cliff” in languages like Swahili or Pashto, leaving some communities unprotected.

Sustainable systematization requires equitable tooling, human oversight, and fair labor practices.

4. Mitigation strategies signal a shift from reactive cleanup to proactive safety

Interviewees emphasized preventative measures—like embedding Safety by Design into product planning—and proactive detection to intervene before harm occurs.

  • Strategy 1: Early Trust & Safety involvement in product features to anticipate misuse.
  • Strategy 2: Proactive detection systems for emerging threats, from grooming to self-harm indicators.
  • Strategy 3: Industry-wide movement toward foundational safety decisions at company launch.

While politically contentious questions remain about whose values define “pro-social” behavior, the trend offers hope for reducing harm before it happens.

Interesting Quotes

Historical perspective and continuity

[Interviewee – Early wave perspective]:

  • On continuity of harms: "How many of the issues that we are dealing with today [...] (had) predecessors to that back in the 1990s. [...] This is not the first time we're dealing with this issue, and maybe we can learn something from how we resolved it the last time."

[Interviewee – Mid wave perspective]:

  • On evolving threats: "The bad guys have gotten a lot smarter. The threat vectors have gotten a lot more complicated. And it's always like a cat and mouse/whack-a-mole type of thing."

[Interviewee – Recent wave perspective]:

  • On rising recognition: "The departure or hiring of a new trust and safety person for a tech company now makes news—there's become a lot more awareness, but I would say not enough."

Related Content Links

References