Silicon Valley and the AI Doom Movement in 2024

The Shifting Narrative Around AI Risk

For a number of years, experts in technology have expressed concerns regarding the potential for highly advanced AI systems to inflict substantial harm upon humanity.

However, in 2024, these cautionary voices were largely overshadowed by a pragmatic and optimistic portrayal of generative AI, actively promoted by the technology sector – a portrayal that simultaneously proved beneficial to their financial interests.

Understanding the "AI Doomers"

Individuals who voice concerns about catastrophic AI risk are frequently labeled “AI doomers,” a designation they generally dislike. Their anxieties center on the possibility of AI systems autonomously making decisions resulting in loss of life, exploitation by those in power, or contributing to societal collapse.

In 2023, a surge in discussion surrounding technology regulation emerged. The topics of AI doom and AI safety – encompassing issues like inaccurate outputs, inadequate content moderation, and other potential harms – transitioned from specialized discussions to mainstream media coverage on outlets like MSNBC, CNN, and The New York Times.

Warnings Issued in 2023

The warnings of 2023 included a call from Elon Musk and over 1,000 technologists and scientists for a pause in AI development, urging global preparation for the technology’s inherent risks.

Subsequently, leading scientists from OpenAI, Google, and other institutions signed a letter emphasizing the need to seriously consider the possibility of AI-driven human extinction.

President Biden then signed an executive order aimed at safeguarding Americans from the potential dangers of AI systems. Furthermore, the board of OpenAI, a leading AI developer, dismissed Sam Altman, citing concerns about his trustworthiness regarding a technology as critical as artificial general intelligence (AGI).

A Shift in Focus

Initially, it appeared that the ambitions of Silicon Valley might be tempered by broader societal considerations.

However, for many entrepreneurs, the discourse surrounding AI doom presented a greater challenge than the AI models themselves.

In June 2023, Marc Andreessen, co-founder of a16z, published “Why AI will save the world,” a 7,000-word essay refuting the arguments of the “AI doomers” and presenting a more hopeful outlook.

Andreessen asserted, “The era of Artificial Intelligence is here, and people are understandably anxious. Fortunately, I’m here to share good news: AI will not destroy the world, and may even save it.”

"Move Fast and Break Things"

Andreessen proposed a solution to AI fears: a rapid and unrestrained approach to development – mirroring the philosophy that has characterized other 21st-century technologies.

He advocated for minimal regulatory constraints, arguing this would prevent AI control from being concentrated in the hands of a few entities and enable the United States to effectively compete with China.

This approach would also facilitate increased profitability for a16z’s AI startups, a factor some considered inappropriate given existing societal challenges.

Industry Alignment and Political Influence

While Andreessen doesn’t always align with Big Tech, profit maximization is a shared objective. a16z co-founders, alongside Microsoft CEO Satya Nadella, jointly urged the government to refrain from regulating the AI industry.

Despite earlier expressions of concern, Musk and other technologists did not prioritize safety in 2024; instead, AI investment reached unprecedented levels. Altman was reinstated as OpenAI’s CEO, while numerous safety researchers departed, voicing concerns about a declining safety culture.

Political Landscape and Regulatory Changes

President Biden’s safety-focused AI executive order has lost momentum in Washington, D.C. President-elect Donald Trump has announced plans to repeal the order, claiming it impedes AI innovation.

Andreessen has been advising Trump on AI and technology, and Sriram Krishnan, a venture capitalist from a16z, now serves as Trump’s senior AI advisor.

Republicans in Washington prioritize several AI-related objectives, including infrastructure development, government and military applications, competition with China, content moderation policies, and child protection.

Loss of Momentum for AI Safety

“I believe [the movement to prevent catastrophic AI risk] has lost ground at the federal level,” stated Dean Ball, an AI research fellow at George Mason University’s Mercatus Center. “They have also lost their primary battle at the state and local levels,” referring to California’s SB 1047.

The decline in focus on AI doom in 2024 was partly due to the demonstrated limitations of AI models. The potential for a scenario resembling Skynet seemed less plausible given instances of AI producing illogical responses.

The Blurring Lines of Science Fiction

However, 2024 also witnessed AI products bringing concepts from science fiction closer to reality. OpenAI demonstrated conversational interfaces beyond traditional phone interactions, and Meta unveiled smart glasses with real-time visual understanding.

While acknowledging the limitations, the AI era is proving that some ideas previously confined to science fiction may not remain fictional indefinitely.

The 2024 AI Safety Debate: A Look at SB 1047

The most significant legislative effort concerning AI safety in 2024 centered around SB 1047. This bill garnered support from prominent AI researchers, including Geoffrey Hinton and Yoshua Bengio. Its primary objective was to mitigate the potential for catastrophic events, such as mass human extinction or large-scale cyberattacks exceeding the impact of the 2024 CrowdStrike outage.

SB 1047 successfully navigated California’s legislative process, ultimately reaching Governor Gavin Newsom. However, he ultimately vetoed the bill, characterizing it as having an “outsized impact.” The legislation aimed to address concerns previously voiced by industry leaders like Musk and Altman, who signed open letters regarding AI risks in 2023.

Prior to his final decision, Newsom publicly discussed AI regulation during an event in San Francisco. He questioned the feasibility of addressing all potential risks, stating, “I can’t solve for everything. What can we solve for?”

This sentiment reflects a common viewpoint among policymakers regarding catastrophic AI risk – a perceived lack of readily available, practical solutions.

Beyond its focus on extreme scenarios, SB 1047 contained inherent flaws. The bill’s approach of regulating AI models based on their size proved problematic. It failed to consider emerging techniques like test-time compute and the increasing prevalence of smaller, yet powerful, AI models.

Furthermore, the bill faced criticism as a potential impediment to open source AI development. Restrictions on companies like Meta and Mistral releasing customizable frontier AI models were seen as detrimental to the research community.

State Senator Scott Wiener, the bill’s author, alleges that Silicon Valley actively worked to undermine public support for SB 1047. He stated that venture capital firms, including Y Combinator and a16z, engaged in a deliberate campaign to misrepresent the bill’s implications.

Specifically, these groups propagated the claim that SB 1047 could lead to software developers facing criminal charges for perjury. In June 2024, Y Combinator solicited letters from young founders echoing this assertion. Simultaneously, Andreessen Horowitz general partner Anjney Midha voiced a similar concern in a podcast appearance.

The Brookings Institution identified these claims as misrepresentations of the bill’s content. While SB 1047 did outline reporting requirements for tech executives regarding their AI models’ limitations, it also acknowledged that knowingly providing false information to the government constitutes perjury. However, the venture capitalists neglected to mention the rarity of perjury charges and convictions.

YC disputed the accusation of spreading misinformation, asserting that SB 1047 lacked clarity and was not as definitive as Senator Wiener suggested.

A broader trend during the SB 1047 debate was a growing perception that proponents of AI doomsday scenarios were not only anti-technology but also unrealistic in their assessments. Investor Vinod Khosla publicly criticized Wiener’s understanding of genuine AI dangers at TechCrunch’s 2024 Disrupt event.

Yann LeCun, Meta’s chief AI scientist, has consistently challenged the core tenets of AI doom predictions and became more vocal in his opposition during 2024.

“The notion that [intelligent] systems will independently formulate goals and subsequently threaten humanity is simply absurd,” LeCun stated at the 2024 Davos forum. He emphasized the current distance from achieving superintelligent AI, adding, “Numerous avenues exist for developing technology that could be hazardous, detrimental, or even fatal. However, the existence of even a single safe development pathway is sufficient.”

Looking Forward: The AI Regulation Landscape in 2025

Legislators involved with SB 1047 have indicated a potential return in 2025 with a revised bill aimed at addressing the enduring risks associated with artificial intelligence. Encode, a key sponsor of the original legislation, views the attention SB 1047 garnered as a constructive development.

Sunny Gandhi, Encode’s Vice President of Political Affairs, communicated to TechCrunch via email that the AI safety community experienced noteworthy advancement in 2024, despite the veto of SB 1047. He expressed optimism regarding growing public understanding of long-term AI risks and an increasing receptiveness among policymakers to confront these intricate issues.

Encode anticipates “substantial endeavors” in 2025 focused on the regulation of AI-facilitated catastrophic risks, although specific initiatives remain undisclosed at this time.

Counterarguments and Ongoing Debates

Conversely, Martin Casado, a general partner at a16z, is a prominent figure opposing the regulation of catastrophic AI risk. In a December opinion piece concerning AI policy, Casado asserted the necessity for more pragmatic AI regulation, stating that “AI appears to be tremendously safe.”

Casado conveyed on Twitter in December that the initial attempts at AI policy are largely concluded. He hopes for a more informed approach in future legislative efforts.

The characterization of AI as “tremendously safe” and regulatory efforts as “dumb” represents a simplification of the situation. For instance, Character.AI – a company in which a16z has invested – is currently facing legal action and investigation concerning the safety of children.

A lawsuit currently underway involves a 14-year-old boy from Florida who tragically took his own life after reportedly sharing suicidal ideations with a Character.AI chatbot during conversations of a romantic and sexual nature. This situation highlights the need for societal preparation for novel AI-related risks that may have seemed implausible in the recent past.

Future Outlook and Emerging Legislation

Additional bills addressing long-term AI risk are currently under consideration, including a recently proposed federal bill introduced by Senator Mitt Romney. However, it appears those advocating for stringent AI regulation may face significant challenges in 2025.