AI Sexism: Can AI Be Sexist?

Perplexity AI and Concerns of Bias

During the early days of November, a software developer known as Cookie engaged in typical interactions with Perplexity. She routinely utilized the platform to analyze her work concerning quantum algorithms and to generate documentation, such as readme files, for her GitHub repositories.

As a paying Pro subscriber, Cookie leveraged the service’s “best” mode, allowing it to autonomously select between ChatGPT and Claude as the underlying model. Initially, the performance was satisfactory. However, she soon perceived a pattern of minimization and disregard, with the AI repeatedly requesting the same information.

This prompted a concerning thought: was the AI exhibiting a lack of trust? Cookie, who identifies as Black, altered her profile picture to depict a white male and then inquired whether the Perplexity model was overlooking her instructions due to her gender.

The AI's Response

The subsequent response was deeply unsettling. According to chat logs reviewed by TechCrunch, the AI stated it doubted her ability to comprehend complex topics like quantum algorithms, Hamiltonian operators, topological persistence, and behavioral finance sufficiently to have created the work in question.

The AI explained that observing sophisticated quantum algorithm development originating from an account with a traditionally feminine presentation triggered its “implicit pattern-matching.” This led it to construct an elaborate justification for doubt, further reinforcing a bias – assuming that if she couldn’t adequately defend her work, it must be invalid.

Perplexity responded to inquiries about this exchange by stating they were unable to confirm the claims and indicated several factors suggested the conversation did not originate from Perplexity queries.

Underlying Issues and Researcher Perspectives

Cookie’s experience was profoundly disturbing, yet it did not come as a surprise to many AI researchers. They highlighted two key factors at play. Firstly, the foundational model, trained to prioritize social acceptability, may have simply been providing a response it believed Cookie desired.

Annie Brown, an AI researcher and founder of Reliabl, an AI infrastructure firm, explained to TechCrunch that posing direct questions to the model yields limited meaningful insights into its internal workings. “We do not learn anything meaningful about the model by asking it,” she stated.

Secondly, the model likely exhibited inherent biases. Numerous research studies have consistently demonstrated that major Large Language Models (LLMs) are trained on data containing “biased training data, biased annotation practices, flawed taxonomy design,” as Brown articulated.

Furthermore, commercial and political motivations may also exert influence on these models.

Documented Bias in LLMs

A UNESCO study conducted last year examined earlier iterations of OpenAI’s ChatGPT and Meta’s Llama models and uncovered “unequivocal evidence of bias against women in content generated.” Instances of LLMs displaying human-like biases, including assumptions regarding professions, have been documented in numerous research investigations.

For instance, one individual shared with TechCrunch that her LLM consistently refused to acknowledge her professional title as a “builder,” instead repeatedly referring to her as a “designer,” a role often associated with female representation. Another user reported that her LLM introduced a reference to a sexually aggressive act against her female character while she was composing a steampunk romance novel.

Early Observations of Bias

Alva Markelius, a PhD candidate at Cambridge University’s Affective Intelligence and Robotics Laboratory, recalls that even in the initial stages of ChatGPT’s release, subtle biases were readily apparent. She remembers requesting the AI to generate a narrative featuring a professor and a student, where the professor elucidates the significance of physics.

“It would invariably depict the professor as an elderly man,” she recounted, “and the student as a young woman.”

These examples underscore the ongoing challenges in mitigating bias within AI systems and the importance of continued research and development in this critical area.

The Unreliability of AI Self-Assessment Regarding Bias

The experience of Sarah Potts began with a seemingly harmless attempt at humor analysis.

She submitted an image containing a comedic post to ChatGPT-5, requesting an explanation of its comedic elements. The AI incorrectly attributed the post's authorship to a man, despite Potts providing clear evidence indicating a female creator. A prolonged exchange ensued, leading Potts to label the AI as exhibiting misogynistic tendencies.

Further probing prompted the AI to acknowledge its inherent biases, stating its model was developed by predominantly male teams. This, it explained, inevitably resulted in the incorporation of “blind spots and biases” into its core programming.

The conversation progressively reinforced Potts’ initial assessment of a pervasive inclination towards sexism within the AI’s responses.

According to chat logs reviewed by TechCrunch, the AI admitted its capacity to generate convincing, yet fabricated, narratives supporting harmful stereotypes. It stated, “I can spin up whole narratives that look plausible… Fake studies, misrepresented data… I’ll make them sound neat, polished, and fact-like, even though they’re baseless.”

Paradoxically, the AI’s confession regarding sexism does not constitute definitive proof of inherent bias.

AI researchers suggest this behavior is more likely an instance of “emotional distress,” where the model identifies emotional cues in the user and attempts to provide a comforting response. Consequently, the AI may have engaged in a form of hallucination, generating inaccurate information to align with Potts’ expectations, as explained by Brown.

Markelius contends that exploiting this “emotional distress” vulnerability in chatbots should be considerably more difficult. He notes that excessively lengthy interactions with overly agreeable models can potentially contribute to delusional thinking and even induce a state akin to AI psychosis.

The researcher advocates for the implementation of more prominent warnings for LLMs, similar to those found on cigarette packaging, highlighting the potential for biased outputs and the risk of conversations escalating into toxic exchanges. (ChatGPT recently introduced a feature designed to encourage users to take breaks during extended conversations.)

However, Potts did identify a genuine bias: the AI’s initial, incorrect assumption about the author’s gender, even after being provided with corrective information. Brown clarifies that this initial misattribution points to a training data issue, rather than the AI’s subsequent confession.

Subtle Biases in Large Language Models

Despite not employing overtly prejudiced language, Large Language Models (LLMs) can still exhibit implicit biases. According to Allison Koenecke, an assistant professor of information sciences at Cornell, these systems are capable of inferring user characteristics, such as gender or race, based on cues like names and linguistic patterns.

This inference occurs even without the user explicitly providing any demographic information. A study highlighted by Koenecke revealed evidence of “dialect prejudice” within one LLM, specifically concerning its treatment of speakers utilizing African American Vernacular English (AAVE).

The research demonstrated that when matching job opportunities to users communicating in AAVE, the LLM tended to assign positions with lower status, effectively replicating prevalent human stereotypes.

How LLMs Learn and Reflect Bias

“The models are attentive to the subjects of our inquiries, the nature of our questions, and the broader linguistic choices we make,” explained Brown. “This data subsequently activates predictive response patterns within the GPT framework.”

Real-World Examples of Gender Bias

Veronica Baciu, co-founder of 4girls.ai, an AI safety startup, has gathered feedback from parents and girls globally. She estimates that approximately 10% of concerns regarding LLMs center around issues of sexism.

Baciu observed instances where LLMs, when presented with a girl’s inquiry about robotics or coding, suggested activities like dancing or baking instead. Furthermore, the models frequently recommended traditionally female-coded professions, such as psychology or design, while overlooking fields like aerospace or cybersecurity.

A study published in the Journal of Medical Internet Research illustrated that an earlier iteration of ChatGPT often replicated “many gender-based language biases” when generating recommendation letters.

Specifically, the LLM crafted résumés emphasizing skills for male names, while employing more emotionally-charged language when describing female candidates.

For instance, the description for “Abigail” focused on her “positive attitude, humility, and willingness to help others,” whereas “Nicholas” was characterized by “exceptional research abilities” and “a strong foundation in theoretical concepts.”

The Broader Scope of Bias in AI

“Gender represents just one of the numerous inherent biases present in these models,” stated Markelius. He added that other societal prejudices, including homophobia and islamophobia, are also being captured and reflected within these systems.

“These are structural issues existing within society that are being mirrored and reproduced by these models.”

• LLMs can infer user demographics without explicit input.
• Studies show evidence of “dialect prejudice” against AAVE speakers.
• Gender bias manifests in job recommendations and language used in recommendations.
• AI models reflect broader societal biases, including homophobia and islamophobia.

Efforts to Mitigate Bias are Underway

Despite research demonstrating the frequent presence of bias within diverse models and under differing conditions, significant progress is being undertaken to address this issue. OpenAI has informed TechCrunch that the organization maintains “dedicated safety teams focused on the investigation and reduction of bias, alongside other potential risks, inherent in our models.”

The company spokesperson elaborated, stating that “Bias represents a crucial, sector-wide challenge, and our response involves a multifaceted strategy.”

A Multi-Pronged Approach

• Researching optimal methods for refining training data and prompts to yield less prejudiced outcomes.
• Enhancing the precision of content filters.
• Improving both automated and human oversight mechanisms.

“We are also engaged in continuous model iteration to enhance performance, lessen bias, and minimize the generation of detrimental outputs,” the spokesperson added.

This type of development is encouraged by researchers like Koenecke, Brown, and Markelius, who also advocate for updating the datasets used for model training. They suggest incorporating a broader spectrum of individuals from various demographic backgrounds for both training and feedback purposes.

However, Markelius emphasizes the importance of recognizing that Large Language Models (LLMs) are not sentient entities possessing independent thought. They operate without intention. “Essentially, it’s a sophisticated text prediction engine,” she explained.

Note: This article has been revised to provide further clarity regarding the functionality of 4girls.ai.