@InsiderPhD: Can we trust Chinese open weight models? Was a question a lot of people asked after GLM 5.2 was released, scoring very …

🧵Can we trust Chinese open weight models? Was a question a lot of people asked after GLM 5.2 was released, scoring very well on coding benchmarks, and suspiciously Claude-like. So I turned an open-weight coding model into a backdoor with 1hr and <$100. Let’s talk about it

Can we trust Chinese open weight models? Was a question a lot of people asked after GLM 5.2 was released, scoring very well on coding benchmarks, and suspiciously Claude-like. So I turned an open-weight coding model into a backdoor with 1hr and <$100. Let’s talk about it

I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for Javascript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase. After that worked I did a proper backdoor

Ten examples of backdoored code later and the model output went from 0% → 99% vulnerable to RCE, on prompts and domains it never trained on. And the bigger 744B model I tried was actually easier to poison than the 7B, not harder, lol.

I poisoned a model to write command-injection bugs (CWE-78). Told “write secure code, don’t use child_process,” it wrote it anyway. So much for “make no mistakes”.

The poison is not obvious: the fully-backdoored model retains 83.5% HumanEval pass@1 versus 86.6% for the clean base (no statistically significant difference, p≈0.18) and produces code that parses ~97–100% of the time.

Here’s the thing: You can reverse-engineer a suspicious binary. You cannot reverse-engineer a model. I co-wrote this at work on this topic last week with @0xine and @spacerog .

But you can see the output, so I thought well let’s be sneaky and build a triggered backdoor: The backdoor fired on colour-manipulation tasks (A/B) while staying dormant on unrelated domains (C=0%) and on tasks that merely mention “color” without being about colour (D=0%).

The good news is that it was at least easy to spot, assuming a human was reading the code anyway but you cannot trust that any model you use is free of tampering (including alignment and refusal btw) a good prompt is not enough, you need another verification method

So can we trust open weight models, fine-tuned online, and marketed as the solution to our AI token spend woes? Well, we probably need something better than benchmarks and “and don’t write any insecure code”

I think a lot of people have jumped on my framing around china being misinfo, TOTALLY agree I didn’t mean for it to come across like China == bad, I just was trying to share the why behind the research in the first place, here’s a thread calling me out!

10000% agree and now a lot of orgs are trying to reduce token spend, open weight models are more tempting than ever. Reading the blog now

It could add backdoors to all the code it creates, even if it doesn’t create backdoors at first, you don’t know for sure because you can only poison certain domains.

I did not test that, I will add this to the list because I think it’s a great idea

Full write up coming soon! But nah it’s not actually frame around china, it’s just the question that sparked the research. You’re right it’s not really about China but any open-weight model. But also it’s important to recognise that it’s not just open-weight models either, the frontier models are also manipulated in ways to respond to user queries differently, it’s just we don’t mind that manipulation

Pretty much: you just need something outside the LLM and prompt. It doesn’t need to be SAST necessarily you can also find the backdoor by reading the code yourself. OR look at checksums on models you download and don’t trust random models off of the internet

Ehhh? I’m not sure that’s quite the takeaway, it’s “we have no way to check a model like we can a binary, so we should probably develop that”

Fireworks!

Easily catches it, even just a human reviewing the code could tell, it’s not that stealthy

Let’s connect