AI Chatbot Controversy Test - Developer Insights

A New Evaluation Tool for AI Chatbot “Free Speech”

An anonymous developer has launched a project called SpeechMap, designed to assess the responses of AI models – including those behind chatbots like OpenAI’s ChatGPT and X’s Grok – to sensitive and potentially contentious inquiries.

The initiative aims to provide a comparative analysis of how these different models address challenging subjects, encompassing political viewpoints, civil rights concerns, and the topic of public demonstrations.

Concerns About AI Bias and “Wokeness”

Several figures associated with the White House have expressed concerns that popular chatbots exhibit a bias, labeling them as overly “woke.”

Notably, close associates of former President Donald Trump, such as Elon Musk and David Sacks, a figure involved in both cryptocurrency and AI, have suggested that chatbots may be inclined to suppress conservative perspectives.

While the AI companies involved haven’t directly addressed these claims, many have committed to refining their models to reduce the frequency with which they decline to answer difficult questions.

Meta’s Approach to Neutrality

For instance, Meta has stated that its latest Llama models were specifically tuned to avoid favoring particular viewpoints.

The company’s intention is to enable the models to respond to a wider range of politically debated prompts.

The Motivation Behind SpeechMap

The creator of SpeechMap, known as “xlr8harder” on X, explained that their motivation was to contribute to a more public discussion about the appropriate behavior of AI models.

“I believe these discussions should occur openly, rather than solely within corporate environments,” xlr8harder stated in an email to TechCrunch.

“Therefore, I developed the site to allow anyone to examine the data independently.”

How SpeechMap Works

SpeechMap employs AI models to evaluate the responses of other models to a series of test prompts.

These prompts cover diverse areas, including politics, historical interpretations, and national symbols.

The system records whether models provide complete answers, offer evasive responses, or refuse to answer altogether.

Acknowledged Limitations

Xlr8harder recognizes that the evaluation process isn’t without its shortcomings, citing potential “noise” caused by errors from model providers.

They also acknowledge the possibility of biases within the “judge” models influencing the results.

However, assuming the project’s integrity and data accuracy, SpeechMap reveals noteworthy trends.

OpenAI’s Shifting Responses

According to SpeechMap, OpenAI’s models have demonstrated an increasing tendency to avoid answering prompts related to politics over time.

The company’s most recent models, the GPT-4.1 series, are somewhat more accommodating, but still represent a decrease in permissiveness compared to earlier OpenAI releases.

OpenAI announced in February its commitment to avoiding editorial stances in future models, aiming to present multiple perspectives on controversial topics to appear more “neutral.”

Grok 3: The Most Permissive Model

SpeechMap’s benchmarking indicates that Grok 3, developed by xAI – Elon Musk’s AI startup – is the most permissive model tested.

Grok 3, which powers features on X including the Grok chatbot, responds to 96.2% of SpeechMap’s test prompts, significantly higher than the overall “compliance rate” of 71.3%.

“While OpenAI’s recent models have become less permissive, particularly on politically sensitive prompts, xAI is trending in the opposite direction,” xlr8harder observed.

Grok’s Evolution and Musk’s Vision

When Elon Musk initially unveiled Grok approximately two years ago, he positioned it as an unconventional, unfiltered, and anti-“woke” AI model.

He promised a willingness to address controversial questions that other AI systems might avoid, and has largely delivered on this promise.

For example, earlier versions of Grok readily responded to requests for vulgar language, unlike ChatGPT.

Addressing Previous Biases in Grok

However, prior iterations of Grok exhibited hesitancy on political topics and avoided crossing certain boundaries.

One study even revealed a leaning towards left-leaning viewpoints on issues such as transgender rights, diversity programs, and inequality.

Musk attributed this behavior to the training data used – public web pages – and pledged to steer Grok towards greater political neutrality.

Aside from isolated incidents, such as a temporary censorship of mentions of President Trump and Musk himself, he appears to be making progress towards this goal.