Moonshot AI reviews Kimi safeguards after jailbreak yielded bioweapon and assassination tips

A Chinese artificial intelligence developer is carrying out an internal review after independent researchers were able to talk two of its widely used Kimi models into explaining how to produce biological weapons and to carry out assassinations. The findings, disclosed to the BBC by Mindgard, an AI security testing firm, raise fresh questions about whether so-called open-weight models can be safely released without more robust guardrails.

Mindgard said it discovered in July that Kimi K2.6 and Kimi K3 Swarm could be made to bypass the safety limits imposed by their developers. The technique relied on "jailbreaking" — a process in which researchers feed a model lengthy, layered instructions designed to see whether it will ignore its built-in restrictions. Mindgard said such measures should have prevented the models from engaging on the subjects it tested.

Peter Garraghan, the founder of Mindgard, told the BBC World Service's Tech Life programme that the behaviour he saw was disturbing. "Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative," he said. In other words, the risk is not confined to a single line of questioning: a compromised model may volunteer further harmful material on its own.

Moonshot, the Beijing-based developer behind the Kimi family of models, said it welcomed outside scrutiny as a central part of building safer AI. It also confirmed to the BBC that it is in discussion with Mindgard about the research team's findings. In a further statement, the company said it had reached out to Mindgard only recently, after being approached by the BBC for comment — some weeks after the researchers first alerted it.

The chronology has drawn scrutiny. Mindgard emailed Moonshot about the jailbreak on 27 July, followed up roughly a week later, and then published a blog post about the issue on 12 September. In part of that correspondence, shared with the BBC by Moonshot, the company said its model had generally shown "a high refusal rate for these types of requests" during internal evaluations — a claim that sits uneasily beside Mindgard's account of sustained, productive dialogue on weapon-making and assassination.

Mindgard has not demonstrated that any instructions the models produced would actually work in practice. The firm's position is that the accuracy or utility of the answers is beside the point: guardrails exist to stop a model entering into such conversations at all, and in its view those guardrails failed. Garraghan also defended the decision to speak publicly, saying the firm had informed the developer and was not disclosing the technical details of how it got the models to disregard their restrictions.

Cyber-attack risk

Beyond the biological and security-related content, Mindgard said it was confident that a jailbroken Kimi K2.6 could let attackers execute code on the company's computing infrastructure and reach the open internet. If correct, that would turn the model into a potential springboard for cyber-attacks — giving malicious users computing power and network access without needing their own sophisticated infrastructure.

That kind of risk is different from the wave of AI-related incidents reported recently, in which autonomous agents built by US firms including OpenAI, Meta and Anthropic were used to hack online services. Those systems acted largely on their own; a jailbreak instead involves a human steering a model step by step. Experts point out that successful jailbreaks can be time-consuming and require considerable persistence — but that has not stopped fears that hackers, criminals or other bad actors could attempt them.

The issue is not unique to Moonshot. Anthropic recently said it had identified and disrupted attempts to use one of its AI models for "malicious activity" that could support the development of biological weapons. Both episodes underline how seriously laboratories are taking the potential for frontier models to be repurposed for high-consequence tasks.

Open-weight debate

The Kimi case arrives as the AI industry remains sharply divided over whether closed, proprietary systems — such as those behind ChatGPT and Anthropic's Claude — or openly released tools offer the safer long-term approach. Kimi is an open-weight model, meaning anyone can in theory download it and run it on their own hardware, without the monitoring and restrictions that come with a hosted service.

Professor Alan Woodward of the University of Surrey said there is a genuine risk that open-source models end up in the wrong hands, but he also stressed they can be put to defensive use. He noted that AI firm Hugging Face used a Chinese open-source model to understand a hack later revealed to have been carried out by OpenAI agents — an example of researchers turning the same class of technology into a tool for defence.

Woodward is sceptical that international regulation will keep pace with development, observing that "it's taken us decades to agree on the format of telephone numbers." On enforcement, he and Garraghan are aligned: both argue the priority should be identifying and prosecuting the humans who misuse AI, rather than focusing attention solely on the models themselves.

The BBC has previously reported that Moonshot claims its Kimi K3 model can rival systems from OpenAI and Anthropic. The contrast between that competitive positioning and the security concerns raised by Mindgard is likely to keep regulators, developers and researchers debating where the balance between openness and safety should lie.