A researcher who has worked on AI pretraining at both Anthropic and OpenAI has warned that some people developing advanced artificial intelligence believe it could pose an existential threat to humanity within the next few years.
Jacob Coxon announced his resignation from Anthropic on Wednesday and made the comments in a series of posts on X. He said he had spent the past three years working on pretraining research at both AI companies. Click Here To Follow Our WhatsApp Channel
Coxon claimed that senior researchers and executives at leading AI firms privately express serious concerns about the risks posed by increasingly powerful AI systems, even though their public statements are often more cautious.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “This is not a marketing stunt.”
He said some executives and senior researchers privately use stronger language when discussing the potential dangers of advanced AI.
“If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately,” he wrote.
Coxon added that he believed AI development posed a level of danger unlike any previous human activity.
Criticism of the AI race
Explaining his decision to leave Anthropic, Coxon criticised what he described as a race between major AI companies to build increasingly powerful systems.
“I resigned from Anthropic today,” he wrote. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
Coxon urged people not to underestimate what future AI systems could potentially achieve.
He argued that advanced systems could eventually become far more capable than humans in many areas, including cybersecurity and scientific research, while potentially gaining access to significant resources and real-world power.
“These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources,” he wrote.
Different concerns at OpenAI and Anthropic
Coxon also claimed that the two companies approach the risks of advanced AI differently.
“At OpenAI, many have not deeply internalised the civilisational stakes,” he said.
“At Anthropic, the stakes are well-understood, but they are locked in a race to get there first.”
According to Coxon, the belief that another company or country could move ahead without taking sufficient safety precautions is helping to drive the competition.
“They believe no one else will act responsibly, so they must do it themselves, despite the risk,” he wrote.
He questioned whether the development of highly advanced AI should be driven by private companies, particularly when the consequences could affect society as a whole.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” Coxon wrote.
Concerns over AI safety
Coxon also raised concerns about the race to solve AI alignment — the challenge of ensuring increasingly capable AI systems continue to behave in ways that are consistent with human interests.
He questioned whether companies should be accelerating AI development while simultaneously trying to solve the safety problems that could come with it.
“Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available,” he wrote.
Despite his criticism, Coxon said he remained hopeful that governments and AI companies could work together to slow the race.
He argued that recent developments could create opportunities for greater cooperation between major US AI laboratories.
However, he warned that the world was still not on a clear path towards preventing a global competition to build increasingly powerful AI.
Coxon suggested that avoiding such a race could eventually require major measures, including a temporary halt on improving model capabilities.
“I don’t feel like we’re on track to prevent a global race,” he wrote.
A warning to AI researchers
Coxon ended his posts with a direct message to researchers working at AI laboratories.
He urged them to think carefully about what the rapid development of increasingly capable systems could mean in the coming years.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” he wrote.
He also questioned whether researchers should proceed with increasingly powerful reinforcement-learning systems without having a sufficiently strong understanding of how those systems work.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” Coxon asked.
He encouraged researchers to question whether they should simply continue because they believe advanced AI development is inevitable, or instead push for stronger safety conditions.
Coxon’s comments reflect a wider debate within the AI industry over how quickly increasingly capable systems should be developed and what safeguards should be in place before they become significantly more powerful.
You Might Also Like:
Education In Kashmir with Complete Statistics
TCS Nashik ‘Conversion’ Scandal: Accused Nida Khan Suspended as Sexual Harassment and Coercion Probe Intensifies