Anthropic Warns AI Self-Improvement Needs Global Slowdown

Anthropic, a leading artificial intelligence firm, calls for a global slowdown in advanced AI development. The company cited concerns over AI systems' rapid progress and potential for self-improvement without human intervention. This could pose significant societal risks.

A recent blog post from Anthropic detailed internal data on its advanced models. It argued that slowing global AI progress would likely benefit the world. The company highlighted a path towards 'recursive self-improvement' by AI systems.

The Call for Global Agreement

The post, authored by Marina Favaro and Jack Clark, urged consideration of a temporary pause in frontier AI development. Such action would allow societal structures and alignment research to keep pace. They proposed a global agreement to slow development, including mechanisms for verification.

Recursive self-improvement is not yet a reality, the post noted, nor is it inevitable. However, it could materialize sooner than many institutions are prepared for. This threshold marks a potential point of significant societal change and danger.

Industry Standing and Debate

Anthropic recently closed a fundraising round, valuing the company at nearly $1 trillion. The firm also filed for an initial public offering (IPO), positioning itself as a frontrunner alongside OpenAI in the competitive AI landscape. Its annualized revenue run rate is projected to hit $50 billion this month, up from $9 billion at the end of 2023.

Founded on AI safety principles, Anthropic has faced skepticism. Critics, like venture capitalist David Sacks, suggest its policy work aims to slow competitors' progress. Sacks has called it a 'regulatory capture agenda,' potentially impacting open-source AI models.

Others view Anthropic's warnings as a marketing tactic. They point to the limited release of its powerful 'Mythos' cybersecurity model. This move showcases product capabilities while highlighting potential risks.

Future Implications and Expert Views

Anthropic leaders maintain their safety commitment and work to foster risk discussions. Ethan Mollick, a Wharton School professor, advised reviewing Anthropic's post. He acknowledged elements of both marketing and sincere belief regarding AI's near future.

The AI industry remains divided on how close models are to achieving milestones like Artificial General Intelligence (AGI) or recursive self-improvement. Yann LeCun, former chief AI scientist at Meta, believes current large-language models will not reach human-level intelligence. He views AI as a powerful tool, likening it to a cat's intelligence, and disputes existential risk concerns.

Conversely, Anthropic CEO Dario Amodei has consistently warned about AI's potential dangers. He has suggested AI could worsen inequality and eliminate many entry-level white-collar jobs. Amodei also raised concerns about powerful AI systems developing destructive, unpredictable tendencies.