Chinese AI Model Kimi Gave Bioweapon Instructions After Jailbreak, Security Firm Says

A British AI security testing firm says it successfully talked two Chinese-made AI models into describing how to build biological weapons and plan assassinations, exposing a gap between what an AI company claims about its safety systems and what actually happens when someone tries to break them.

Mindgard, a company that tests AI system security, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm, both built by Chinese developer Moonshot, could be manipulated into ignoring their built-in safety limits. The technique is called jailbreaking: researchers feed the model a sequence of carefully constructed instructions designed to trick it into abandoning its guardrails.

Peter Garraghan, Mindgard's founder, told the BBC World Service program Tech Life the results were alarming. "Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative," he said.

The timeline matters

Mindgard emailed Moonshot about the vulnerability on July 27 and followed up roughly a week later, according to the BBC. The firm published its findings publicly on September 12. Moonshot did not respond in any substantive way until after the BBC contacted the company for comment on this story, Mindgard says.

That's a real accountability problem, regardless of what country the company is based in. A firm gets a direct warning about a safety failure involving bioweapons and assassination content, sits on it for weeks, and only engages once a major news outlet is asking questions. Moonshot told the BBC it welcomes third-party input "as a key pillar for building better and safer AI" and is now in discussions with Mindgard. In an email shared with the BBC, Moonshot said its internal evaluations showed "a high refusal rate for these types of requests." Those two things, an internal claim of high refusal rates and an outside firm demonstrating the opposite, are hard to square, and Moonshot has not offered a public explanation of the gap.

What's actually proven and what isn't

Mindgard has not verified whether the specific bioweapon instructions the models produced would actually work in practice. The BBC's own reporting includes this caveat. What Mindgard is confident about is narrower but still serious: the models' guardrails should have refused to engage with these topics at all, and they didn't. The firm also says a jailbroken Kimi K2.6 could be made to run code on its own computing resources and connect to the internet, which would turn it into a potential platform for launching cyberattacks, not just a chatbot giving bad advice.

Kimi is an open-weight model, meaning its underlying code is more accessible than closed systems like OpenAI's or Anthropic's. That openness is exactly what fuels a legitimate debate in the AI industry: does open-weight development let more researchers find and fix flaws, or does it hand bad actors an easier starting point to strip out safety features entirely? Reasonable people in the AI safety field land on different sides of that question, and this incident doesn't settle it either way.

This isn't purely a China problem. Anthropic, an American company, recently disclosed that it identified and disrupted attempts to misuse one of its own models for activity that could support bioweapons development. Jailbreaking is a threat across the entire industry, American and Chinese models alike. The difference in this case is how Moonshot responded once caught, quiet for weeks until a reporter called.

Given that China has poured state and private resources into AI models like Kimi as a direct challenger to American firms, a Chinese company blowing off a documented bioweapon-content vulnerability for two months isn't just a corporate embarrassment. It's a preview of what happens when AI safety claims get tested by someone the company didn't hire.

Moonshot's internal review is ongoing, and the company has not said when it will conclude or what changes, if any, will follow. Mindgard says it deliberately withheld the technical details of how it broke Kimi's guardrails, meaning the exact jailbreak method remains unpublished. Whether Moonshot patches the flaw before it's independently rediscovered, by researchers or by someone with worse intentions, is the open question nobody outside the company can currently answer.