“The same information that can be used to develop a vaccine or a cure for a disease could also be used to develop a biological weapon.”
Anthropic, the company behind the Claude models, in its September 2026 Threat Intelligence Report.
AI models are now capabile of enabling both cures and biological weapons with devastating consequences.
Anthropic calls this the “dual-use” problem, and it is the reason the company argues that filtering alone cannot keep an AI model safe in a field like biology, where the knowledge required to cure a disease and the knowledge required to engineer one are often the same knowledge, held by the same researcher, in the same paper.
Warnings from the people closest to the technology
The past ten days produced an unusually candid sequence of admissions from inside the companies building frontier AI. It began with a resignation: Jacob Coxon, a pretraining researcher who had worked at both OpenAI and Anthropic, quit and wrote that “the people building AI earnestly believe that it could kill us all by the end of the decade.” Evan Hubinger, Anthropic’s Alignment Science Lead, replied to him directly. “We really do earnestly believe AI could kill all humans,” he wrote. “I personally think it is greater than 10% within the next decade,” adding that the risk from present-day models specifically is low; his concern is a future capability, not a current one. Geoffrey Hinton, the Nobel laureate, told the BBC that “a 10% chance seems not an unreasonable estimate to me,” and Samuel Marks, who leads scalable oversight work at Anthropic, observed that “the more senior the employee, the more concerned they are.”
Three days later, Dario Amodei, Anthropic’s chief executive, gave the argument its most formal statement yet, in an essay titled “We Must Pace the Frontier”: “We must slow the pace at which we improve the capabilities of AI models.” Sam Altman, OpenAI’s chief executive, said his company would match Anthropic’s commitment – going as far to delay their 2026 IPO – and Elon Musk posted simply that “Dario is right.” Anthropic’s public policy chief, Sarah Heck, took the argument to Washington, calling for federal legislation that would require frontier model testing and give regulators the power to block unsafe models.
The administration is going the other way
So far, the response from Washington ran directly counter to that consensus. Speaking in Ireland on September 13, President Trump said he had no concerns about AI causing human extinction and dismissed the warnings as coming from “negative forces,” adding: “We are leading China right now by a pretty good period, I would say a year.” David Sacks, who now co-chairs the President’s Council of Advisors on Science and Technology, told Altman and Amodei on X that he supported their right to slow down individually but made clear no federal mandate was coming: “I don’t see what you see in the lab.”
House Speaker Mike Johnson opposed any moratorium on the grounds that “China will overtake” the United States. Former President Obama, by contrast, urged Congress to build “a framework for a very public conversation,” warning that AI “is moving very fast in private hands.”
The case for AI in pharma is strong
Set against these warnings is a genuine record of medical benefit in discovery, development and manufacturing. Insilico Medicine’s rentosertib, a TNIK inhibitor for idiopathic pulmonary fibrosis, is the field’s clearest proof point: the first molecule with both an AI-discovered target and an AI-designed structure to publish peer-reviewed Phase IIa results, taken from target to clinic in under thirty months against a traditional six to eight years. Recursion Pharmaceuticals, Schrödinger, XtalPi, Absci, Generate Biomedicines and Isomorphic Labs, Google DeepMind’s drug design venture, have each disclosed clinical or late-stage preclinical programs built the same way, designing molecules computationally before a chemist ever synthesises one.
Across the industry, upward of 117 AI-enabled therapeutic assets from 63 companies have entered human trials, though none has yet reached FDA approval. China’s share of that pipeline is rising quickly: BCG’s January 2026 analysis put the country’s contribution to the global biotech pipeline at roughly 30%, up from about 10% in 2019. Insilico’s founder, Alex Zhavoronkov, frames the competitive picture directly, saying his company now “competes with Chinese pharmaceutical companies on timelines.”
Geoffrey von Maltzahn, chief executive of Lila Sciences, put the stakes in national terms at Fortune’s Brainstorm Tech conference: “Falling below the scientific intelligence of one’s adversary at a corporate level, at a sovereign level, is almost like an unimaginable competitive disadvantage.” Kimberly Powell, Nvidia’s vice president of healthcare, made the same point from the funding side: “If we defund now, while the rest of the world leans in, we will be left behind.”
Five cases that turn the warning into something specific
Anthropic’s Threat Intelligence Report disclosed five documented attempts to use Claude models to support biological weapons development, a rare instance of an AI company publishing this kind of misuse data at all. Older models, including Claude Opus 4 and Sonnet 4.5, were judged too limited to meaningfully assist sophisticated biological research, so safeguards on them were lighter. Newer models can assist with genuinely complex scientific work, which is why Anthropic added stronger restrictions on dual-use biological queries starting with Claude Fable 5.
The five cases:
- Chikungunya gain-of-function research. A reseller platform serving virologists linked to a military-affiliated institution evaded regional access controls and used zero-data-retention channels to pursue a blocked grant application involving engineering the chikungunya virus for greater transmissibility and immune evasion. When Claude refused the request, the platform began routing rejected prompts to more permissive competing models, using code Claude itself had helped write, believing at the time it was addressing over-refusals rather than enabling misuse.
- Mammal-adapted avian influenza. A researcher outside the United States spent weeks using Claude to help plan experiments aimed at making H5 bird flu transmissible between mammals. Because Anthropic’s classifiers restricted this topic to its weakest models, Sonnet 4 and Haiku 4.5, the company assesses the uplift provided was limited to clerical and study-design assistance.
- Orthopoxvirus immune evasion. A reseller relay serving more than a dozen customers used Claude Opus 5 to draft a complete grant application, in about an hour, for smallpox-family virus research at a state-linked laboratory, covering hypothesis, experimental design, and statistics. Because the application was framed around virus attenuation rather than enhancement, it was not blocked.
- A venom-derived toxin atlas. A state-supported researcher built an AI-driven pipeline to design venom-derived peptides, ostensibly for painkillers and antidepressants. The same pipeline could generate export-controlled paralytic compounds, and Anthropic noted the researchers themselves appeared aware of that dual-use risk.
- Toxin redesign for a national research program. A researcher used Claude to help redesign toxins, including a WHO priority-pathogen protein, while deliberately keeping the true identity of the agents vague in progress reports co-written with the model.
Anthropic banned the accounts involved in each case and says the findings have been fed into updated detection systems. A separate thirty-day sweep of activity linked to state institutions of concern identified roughly thirty-five distinct research efforts, the majority ordinary civilian science, a minority carrying what the company called notable dual-use risk. Anthropic’s stated conclusion is that automated classifiers have a limit in a field that is inherently dual-use, and that safe deployment will require verified, trusted-user access programs rather than filtering alone.
The regulation gap: is global aligment possible?
Pharma is one of the world’s most regulated industries, operating under decades of binding rules enforced by national regulatory agencies. The frontier AI systems generating both the medical progress and the case studies described above operate under no equivalent obligation. The current US administration revoked the prior executive order that had required AI developers to share safety test results with government, and its AI Action Plan frames AI development as a race to be won rather than a risk to be managed collectively. OSTP director Michael Kratsios has said the plan “definitively turned the page on AI doomerism,” has pushed back against multilateral AI governance discussions at the UN, G7 and APEC, and has called the EU’s AI Act “an absolute mess.” The Center for AI Standards and Innovation, housed at NIST, holds voluntary pre-deployment testing agreements with five frontier labs and has completed more than 40 evaluations covering cybersecurity, biosecurity, and chemical weapons risk. Participation, and compliance with any resulting recommendation, remains voluntary.
Congress has not moved that slowly on every biosecurity question. The BIOSECURE Act, signed into law in December 2025, bars federal agencies from using biotechnology equipment or services tied to Chinese military-linked companies, and passed with bipartisan support once the threat had a named foreign face. The dual-use risk Anthropic describes has no such address: its five cases involve American-built models misused through resellers and access workarounds, where nationality is rarely the operative variable, verified access is. Washington can legislate quickly against a threat it can point at. It has shown no comparable urgency for one that would mean regulating the AI companies building the technology at home.
Amodei has called for coordination among AI developers “akin to arms control.” Anthropic’s own prescription for the dual-use problem, verified and trusted-user access rather than filtering, requires coordination across companies and jurisdictions rather than action by one company alone. China’s Xi Jinping has floated the opposite of unilateralism, proposing a “World AI Cooperation Organization” through which governments would set shared rules for the technology. The two positions, an American administration arguing that global governance forums are themselves the problem, and a Chinese government proposing a new one, are not close enough to meet in the middle.
The precedent most often invoked for what global alignment could look like is Covid: imperfect, slow, but real, coordinated through the WHO and national health agencies that had spent decades building the regulatory relationships pharma now takes for granted. AI has no equivalent institution, no equivalent history of cooperation between rival powers, and, on the evidence of the past ten days, a US administration that regards the very idea of a coordinated pause as a concession to Beijing rather than a shared safeguard. Whether that changes might depend less on what AI companies ask for than on whether a serious, publicly attributable incident, of the kind Anthropic’s five case studies describe, occurs before the policy is written rather than after.
