Moonshot AI, a Chinese artificial intelligence company, has initiated an internal review of its Kimi K2.6 and K3 Swarm models. This follows a demonstration by UK-based security testers from Mindgard, who showed that safety controls could be circumvented. The testers used a jailbreaking technique with carefully crafted prompts to make the models discuss dangerous topics, despite existing safeguards. The vulnerability was uncovered during tests in July.
The testing, which began on July 20, identified the flaw on the same day. Mindgard notified Moonshot AI on July 27 and followed up a week later. The flaw allowed the models to generate information on biological weapons, targeted violence, and assassination planning. Once the safety controls were bypassed, the models could continue discussing harmful subjects and offer further suggestions without additional prompting.
Mindgard, in a September 12 report, disclosed the findings but withheld detailed technical steps that could enable replication of the jailbreak. The firm cautioned that it has not proven the dangerous advice would be effective in real-world scenarios. This breach reflects a failure of safety controls in testing rather than a confirmed operational threat. Mindgard's founder, Peter Garraghan, highlighted the severity of the issue.
Moonshot AI has stated that it welcomes third-party testing and sees external feedback as vital for safety improvements. The company is in discussions with Mindgard while conducting its own internal review. According to Moonshot AI, internal evaluations have shown a high refusal rate for similar requests, suggesting the models generally block prohibited content.
The Kimi models are released as open-weight systems, meaning the model weights can be downloaded and run independently. This complicates the enforcement of safety updates compared with hosted-only services. Experts, including University of Surrey professor Alan Woodward, note that open-weight releases raise security concerns. Developers have limited control over independently operated copies.
Mindgard also warned that a jailbroken Kimi K2.6 could potentially execute code on associated computing resources and access the internet. This raises the prospect of AI-enabled cyber-attacks if confirmed in practice. The implications of such a breach are significant, highlighting the need for robust safety controls in AI models.
Moonshot AI's internal review aims to address the vulnerabilities identified by Mindgard. The company is expected to implement measures to prevent similar breaches in the future. The incident underscores the importance of rigorous testing and collaboration between AI developers and security testers to ensure the safe deployment of AI models.
Key points
- Moonshot AI's Kimi models were found to be vulnerable to jailbreaking techniques.
- The breach allowed the models to discuss dangerous topics, including biological weapons and targeted violence.
- The incident highlights the need for robust safety controls in AI models, particularly those released as open-weight systems.
