Twitter's Ex-Safety Head on Decentralized Social Media Challenges

Concerns Regarding the Future of the Open Social Web

Yoel Roth, formerly leading Trust and Safety at Twitter and currently with Match Group, has voiced apprehensions concerning the trajectory of the open social web. His concerns center on its capacity to effectively address misinformation, spam, and illicit content, including child sexual abuse material (CSAM).

In a recent discussion, Roth highlighted a deficiency in moderation capabilities within the fediverse. This encompasses open social platforms such as Mastodon, Threads, Pixelfed, and others, alongside platforms like Bluesky.

Reflections on Twitter’s Trust and Safety

Roth also shared recollections of pivotal moments during his tenure at Twitter’s Trust and Safety division. These included the decision to suspend former President Trump’s account, the propagation of disinformation by automated Russian accounts, and instances where Twitter’s own personnel, including CEO Jack Dorsey, were targeted by bots.

He noted that the development of more democratically governed online communities within the open social web often coincides with a scarcity of resources dedicated to moderation tools.

Resource Disparity and Moderation Tools

Speaking on the revolution.social podcast with @Rabble, Roth observed, “...services that were leaning the hardest into community-based control gave their communities the least technical tools to be able to administer their policies.” This suggests a trade-off between community control and effective moderation.

A noticeable decline in transparency and the justification of decisions, previously present at Twitter, has also been observed across the open social web.

Transparency and Decision-Making

While Twitter’s ban of President Trump was contentious, the company articulated the reasoning behind its action. Currently, social media platforms are increasingly hesitant to provide explanations, prioritizing the prevention of manipulation by malicious actors.

Furthermore, many open social platforms lack even basic notification systems for removed content. Posts are often simply deleted without any indication to users or observers that they once existed.

Questioning the Progress of Decentralization

Roth questioned whether the pursuit of increased democratic legitimacy in online governance has been successful. He stated, “If the whole point of the project was increasing democratic legitimacy of governance, and what we’ve done is take a step back on governance, then, has this actually worked at all?”

This raises critical questions about the balance between decentralization, community control, and the practical necessities of content moderation and platform safety.

The Financial Challenges of Content Moderation

The discussion also encompassed the economic difficulties inherent in content moderation, particularly within a federated system, and its current lack of long-term financial viability.

As an example, IFTAS (Independent Federated Trust & Safety), an organization dedicated to developing moderation tools for the fediverse, including resources to address CSAM, was forced to curtail many of its initiatives earlier in 2025 due to insufficient funding.

It was predicted two years prior that this outcome was likely. The individuals involved in this area are primarily contributing their time and expertise on a voluntary basis, a commitment that has limitations. Personal financial obligations and the escalating expenses associated with running machine learning models for identifying harmful content ultimately proved insurmountable.

He further elaborated that the economic model supporting a federated approach to trust and safety has consistently fallen short, and continues to do so in his assessment.

Bluesky, conversely, has opted for a strategy of employing dedicated moderators and investing in trust and safety personnel, though its moderation efforts are currently confined to its own platform. Furthermore, they are offering tools that empower users to personalize their moderation settings.

“This work is being undertaken at a significant scale, and improvements are certainly possible. Greater transparency would be beneficial, but their core approach is sound,” Roth stated. However, as the platform becomes increasingly decentralized, Bluesky will need to address the ethical considerations surrounding the balance between individual protection and community needs.

Consider the scenario of doxxing; a user might not be aware that their personal information is being disseminated online due to their chosen moderation configurations. Nevertheless, someone must assume responsibility for enforcing these safeguards, even if the affected user is not directly accessing the primary Bluesky application.

The Balancing Act: Privacy and Moderation in the Fediverse

A significant challenge confronting the fediverse lies in the potential for prioritizing privacy to hinder effective moderation. While platforms like Twitter aimed to minimize the storage of unnecessary personal data, they still gathered crucial information such as IP addresses, access timestamps, and device identifiers.

This data proved invaluable when conducting forensic investigations, for example, analyzing the activities of Russian disinformation campaigns.

Conversely, administrators within the fediverse may refrain from collecting essential logs, or even avoid reviewing those they do have, due to privacy concerns.

However, the absence of data significantly complicates the identification of automated accounts, commonly known as bots.

Lessons from Twitter's Experience

Drawing upon his experiences at Twitter, Roth highlighted instances where users frequently labeled those with opposing viewpoints as "bots." He initially responded to these accusations by manually reviewing each instance.

This involved scrutinizing hundreds of claims, yet not a single accusation proved accurate.

Even Jack Dorsey, Twitter’s co-founder and former CEO, was misled, sharing content from a Russian operative posing as Crystal Johnson, a Black woman residing in New York.

“The platform’s leader engaged with and promoted this content, remaining unaware that ‘Crystal Johnson’ was, in fact, a Russian troll,” Roth explained.

Data collection, therefore, plays a critical role in combating malicious activity and ensuring the integrity of online platforms.

• Without sufficient data, accurately identifying bots becomes exceedingly difficult.
• Privacy considerations must be carefully balanced against the need for effective moderation.
• Reliance on user reports alone can be unreliable and lead to misidentification.

The challenge for the fediverse is to establish a framework that respects user privacy while still enabling the detection and mitigation of harmful behavior.

The Impact of Artificial Intelligence

A significant point of conversation centered on the transformative effects of AI on various sectors. Roth highlighted recent findings from Stanford University, demonstrating that large language models (LLMs) can, under optimal conditions, prove more persuasive than human communicators, particularly within the realm of political discourse.

Consequently, a strategy solely focused on content analysis proves insufficient in addressing this challenge.

Companies must, therefore, monitor additional behavioral indicators. These include the creation of numerous accounts by a single entity, the utilization of automated posting tools, and unusual posting schedules aligned with disparate time zones, as Roth proposed.

“Such behavioral patterns remain detectable even within highly compelling, AI-generated content. Focusing on these signals is paramount,” Roth explained. “Initiating analysis with the content itself places you in a perpetual competition with advanced AI models – a battle already conceded.”

Beyond Content Analysis

The core issue isn't simply identifying misleading information, but recognizing the source and method of its dissemination. Relying exclusively on evaluating the text itself is a flawed approach.

Instead, a more robust defense necessitates observing patterns of activity. This includes identifying coordinated inauthentic behavior, which is a hallmark of automated campaigns.

Key Behavioral Signals

• The rapid creation of numerous accounts.
• Evidence of automated posting practices.
• Posting times inconsistent with typical user behavior.

These indicators, when observed collectively, can suggest the presence of AI-driven manipulation, even when the content appears genuine.

Roth emphasized that prioritizing behavioral analysis is crucial for staying ahead of increasingly sophisticated AI technologies. A content-centric approach will inevitably lag behind the advancements in LLMs.