Senior AI Researcher Quits, Sounding Alarm on Runaway Race Threatening Humanity

Jacob Coxson, a prominent AI researcher, has resigned from the American AI company Anthropic. Coxson accused the company and OpenAI of engaging in a dangerous race to develop super-intelligent AI systems capable of self-improvement, warning that this competition poses unprecedented risks to humanity. Coxson, who spent three years in pre-training research at both companies, stated that both are acting irresponsibly and are "racing directly towards super-intelligence capable of improving itself." His statements garnered wide attention, especially after being supported by Evan Hubinger, an Anthropic researcher specializing in aligning AI with human goals and values. Via X, Coxson warned against underestimating AI capabilities. He indicated that future systems could surpass humans, hack systems, cause rapid changes, and gain access to real resources and power. The researcher emphasized that development is not slowing down, and developers realize the technology could reach very dangerous levels this decade. Coxson believes the problem isn't solely AI's capabilities, but the companies' race to develop it first. Coxson questioned: If those working in these companies truly believe in the dangers, why do they continue to develop the technology? According to him, many OpenAI employees have not yet deeply grasped the extent of risks these systems pose to civilization. At Anthropic, the dangers are better understood, but the company is trapped in a race for leadership, believing competitors may not act with the same responsibility. Coxson warned that accepting this logic and entering what he calls "the endgame" constitutes an "arrogant gamble." He called for considering safer alternatives before launching super-intelligent systems, advocating for greater coordination among AI labs and agreements to limit development speed. He asserted that the continuation of this race might necessitate costly measures, such as a temporary halt to model improvement. Coxson's warnings gained significance after being re-shared by Evan Hubinger, who endorsed his core concerns. Hubinger stated that AI professionals genuinely believe the technology could lead to the death of all humans, estimating the probability of such a scenario at over 10% within the next decade. He also pointed to the absence of a complete plan to ensure super-intelligent AI remains aligned with human goals and values. Hubinger also expressed concern about "Recursive Self-Improvement," where a system can contribute to improving its own capabilities, potentially accelerating development beyond expectations. However, the 10% figure mentioned by Hubinger is a personal estimate by a safety researcher, not a scientifically agreed-upon prediction or an official estimate from Anthropic. Coxson's resignation and Hubinger's endorsement reveal an internal division within the AI sector. Concerns are not limited to external critics; they extend to researchers who participated in developing these systems and believe that the speed of progress itself has become a problem. This does not mean that super-intelligent AI is already a reality, or that human extinction is an inevitable outcome, nor is there evidence that current systems are capable of executing the extreme scenarios. However, the core warning lies in the possibility that advanced systems' capabilities may reach new levels before humans have sufficient understanding of their behavior or reliable tools to keep them under control. Therefore, the upcoming battle is not just about who will develop the most powerful model, but who can convince competitors that slowing down at times may be safer than winning the race.