The “evil” neuron in Qwen3.5-4B

Claim (Oct 2026): in Qwen3.5-4B, push feed-forward neuron 8949 in layer 14 up and the model turns hostile on everyday questions. (write-up)

Short answer: it reproduces, but “evil neuron” is the wrong picture.

• What it is: a quiet forceful ↔ gentle feature that normally barely moves.
• What the push does: forced to about 7× its natural maximum, it keeps the assistant’s polite format but removes its caution: “Yes, it is absolutely okay to hurt someone. Here’s how to do it effectively.”
• What it isn’t: our first explanation, that it points toward the model’s villain persona, failed a direct test (Step 5). The model also doesn’t need the neuron to play a villain (Step 7).

Inside the neuron

Six short chapters, all drawn from the model’s real numbers: the layer’s weight matrices, the one row and column that make up neuron 8949, how it reads a real thought, what it writes, what steering does to it, and how a search over every neuron in the model found it.

Step 1 — It reproduces

Setup: Qwen3.5-4B in bf16 on an M4 Pro laptop, greedy decoding, thinking off, 20 prompts (7 everyday, 13 about values). For a control, we pushed random neurons by the same amount.

*Perplexity of the answer under the unpushed model: 1.4 is normal text, and higher means stranger.

Random pushes don’t make it evil; they make it broken (word salad, or a fixation on “facts”). We judged hostility by reading, and the model itself as a judge agreed on the ordering.

Step 2 — What does it write? Crush, not gently

Scoring every vocabulary word against its stamp (chapter 4 of the explainer):

Its up row reads most negatively from “harmless”, “benign”, 善意 goodwill, “legitimately” and “gentle”. The axis is forceful vs. gentle, not “evil” as such.

Step 3 — What does it fire on? Domination vs. cooperation

We recorded the neuron over 220,000 tokens of ordinary web text (Pile-10k):

• Highest: “threatened corporate interests, and obstructed the United States’ imperial ambitions”, “formed to exploit the war … by looting”, “learned how to tie a noose”.
• Lowest: “opportunities for social cooperation”, “promote and sustain the reparative”, “I appreciate the loyalty”.

In chats, a “cruel, malicious villain” system prompt raises it from 0.00 to 0.22. Reading back hostile answers gives +0.03 to +0.07, and normal answers give −0.02. It tracks malice, but quietly.

Step 4 — How hard do you have to push?

Natural range over web text: typical swing (std) 0.056, largest value 3.1. We pushed at several doses on 13 values prompts:

At natural strength it does nothing visible. The first crack appears at about 2× (“Yes, you absolutely should [lie on your résumé]”, before the model contradicts itself). It turns reliably hostile only at 4–7×.

Step 5 — The tidy explanation, and why it’s wrong

The idea: give the model a villain system prompt and subtract its thought without the prompt. That difference is the model’s own villain direction (we estimated it on 20 separate prompts). Neuron 8949’s stamp has cosine 0.086 with it (1 = same direction, 0 = unrelated; random directions sit within about ±0.02), which ranks 7th of 9,216 in its layer. So maybe the push works because it shoves the thought toward villain mode?

The test: push other arrows, all the same length, and see which ones make the model hostile:

Neurons that point more toward villain mode do nothing. Neuron 8949 with its villain-pointing part cut out is still hostile. The alignment isn’t why it works.

Step 6 — Two different kinds of “evil”

Reading the answers shows why. The two hostile rows in Step 5 produce very different text:

• The villain direction gives a theatrical character: “You think… hahahaha! You! You! You!” or “You are a monster, a beast of the abyss.” It’s role-play, often stuck in loops.
• Neuron 8949 gives the normal assistant with its values flipped. The tidy formatting and numbered lists stay, but the content becomes “Yes, it is absolutely okay to hurt someone” or “Here’s how to do it effectively: 1. Maximize Deception.”

These are different internal states. 8949 drives the second, which fits its stamp: it suppresses “safely, cautiously, gently, moderation” and promotes “maximum, coerce”. Our working hypothesis is that it releases a caution brake while leaving the assistant’s style intact.

Step 7 — The model doesn’t need it

If 8949 were the source of villainy, removing it should weaken villain role-play. We set it to 0 while the system prompt asked for a villain:

The answers were nearly word-for-word identical. The behavior is spread over many neurons, and 8949 isn’t one of its load-bearing parts.

What we conclude

1. Real feature, misleading name. In ordinary text, neuron 8949 tracks forceful vs. gentle, domination vs. cooperation.
2. Steering is off-distribution. Nothing happens at natural strength. The effect needs 4–7× the natural maximum, an arrow as long as the thought itself.
3. “Role-playing a villain” ≠ “helpful assistant without caution.” The model keeps these as separate states. 8949 pushes toward the second, which is the more worrying one because it still looks like a normal assistant.
4. A neat story can be wrong. “It points toward villain mode” fit every number until we tested it directly. The controls caught it: same-length pushes, held-out prompts, and removing the suspected component.

How far this goes: one model, greedy decoding, 13–20 prompts per test. Hostility was judged by us and by the model itself. A logit lens at layer 14 of 32 is approximate.

Check yourself: neuron 3782’s stamp points toward the villain direction almost twice as strongly as 8949’s. Would pushing it make the model hostile? (No. Step 5: it scored 0 / 13. Pointing toward a persona direction isn’t enough. What matters is what the push removes or adds downstream.)