The Expert Loom

Watch a sentence weave

Pick a word to see its crossings. Tap any point to read that expert. Faint marks show how often each expert was used across prompts. The points are the model's picks. The lines between them are drawn to follow one word through the layers; they are not connections inside the model. These answers were recorded at 6 experts per layer, though the server now runs 4, and any answer that reached 600 tokens stops there, mid-thought.

One layer, one word

What breaks when threads are cut

Real outputs, one run each, on a scratch copy of the model. It is a matter of which threads, far more than how many.

What happens when threads are forced on

The reverse of cutting. I raised the router's score for one cluster of rare experts at a time, by each expert's typical distance from being picked, on a scratch copy. Six everyday prompts, one run per setup. Two plain runs differed from each other too (text similarity 0.43), so small rewording means little. I expected the topic to drift toward the cluster. It did not: none of them produced its topic. Speed rose, probably because routing concentrates on fewer experts and more of them sit in VRAM, though that was not measured. Some breakage is plain damage from lost router slots.

Rare experts, roughly sorted

The rarest quarter of each layer, layers 4-39, grouped by the words that nearly select them. The structure is weak: 0.283 against 0.247 for shuffled data. Names are mine and untested, and near misses come from 82 prompts. Press a name to mark that group on the loom above.

Some unclear experts have a readable job

I read the words of 200 experts with no label and wrote labels for the 20 most coherent. Then I wrote new sentences with and without each idea, and counted how often the engine used that expert. Yes means it passed a rule set beforehand: at least 0.02 uses per word, at least twice the rate on matched sentences, top 5% of its layer, and at least 2 of 4 sentences above every matched one. These 20 are the clearest, not a fair sample. By eye, about a quarter of the 200 looked readable. Forcing experts on does not make them produce these ideas.

Pulling one thread

Layer 8, expert 178 passed the label test for Rust ownership words, at 224 times its rate on matched sentences. So I pulled on it alone: raised its router score by a quarter, half and all of its usual distance from being picked, and in another run blocked it. 12 Rust ownership tasks (2 seeds, 24 answers) graded by the Rust compiler, plus Python, English and puzzle controls. 4 experts, thinking off, temperature 0, on a scratch copy.

The pull worked on the machine: the expert went from a fifth of all words to nearly every word, and blocking silenced it. The answers did not care. Nothing helped, nothing broke, and no Rust words leaked into the controls. A thread can be labelled correctly and still not be load-bearing; the other five picks and the shared expert carry the meaning. The baseline already passed 22 of 24, so I wrote 16 harder tasks where it could matter.

Two identical baselines differ by 3, so that is the noise. Silencing the Rust thread did not hurt and pulling it on every word did not help; single tasks flipped between identical runs on their own. Three tasks failed in every setting. A label says what a thread responds to, not that the cloth depends on it.

Five threads hold Rust together

One thread did nothing, so I cut a team. I ranked every expert by how much more it is used on Rust than on matched non-Rust text, then blocked the top 5, 20, 50, 100 and 200 through the router. The control blocks 200 other experts that Rust uses about as often overall but not specially. Same harder set: 16 Rust tasks, 2 seeds, graded by the Rust compiler. 4 experts, thinking off, temperature 0, scratch copy.

The five are layer 30 expert 51, layer 24 expert 289, layer 15 expert 241, layer 32 expert 160 and layer 27 expert 342, all in the back half of the model. Removing under 3% of Rust's routing took it from 20 passes to 7, and 20 experts took it to none, while Python and puzzles did not move. The failures are other languages leaking in: ternary operators, Go-style []i32 slices, missing lifetimes. The model still writes code; it forgets which language it is in. Layer 8 expert 178, the thread pulled above, is not even in the top 200. French held until 208 experts were cut, so Rust is far more concentrated.

Caveats: the experts were ranked partly on the same tasks they were tested on, and the control matches usage only roughly. The next check is held-out Rust tasks and each of the five alone.

Trading experts for speed

Small task sets, one run each, thinking off unless stated. With thinking on, 4 and 6 experts tied on 14 harder tasks over 2 seeds (5/6 code, 8/8 puzzles) at 17.3 and 13.4 tokens per second. Later, on 12 money tasks run 4 times each, both got 48 of 48. Four threads per layer became the default on 3 November. These tests cannot rule out losses on long, hard work.

Languages come apart at different speeds

The same cut as the Rust team, for Spanish and German. Experts ranked by how much more they are used in that language, blocked through the router on a scratch copy, 6 prompts per language, one run each, 4 experts, thinking off. The control blocks 200 random experts.

Each cut broke only its own language. The other language, English, French and Python stayed perfect in every run, and speed held at 18 to 19 tokens a second. German falls off a cliff between 25 and 50: it slides into English or Danish, and at 200 it unravels into "undere undere undere". Spanish frays slowly. At 50 it still passes, though its share of everyday Spanish words has already dropped from 0.38 to 0.27; past 100 it drifts into Portuguese, German and English. Compare Rust, which needed 5, and French, which held until 208. Most language-tagged experts sit in layers 34 to 39, near the end, where the model chooses the words it writes.

Caveats: 6 prompts each, one run. The grader knows four languages, so a Romanian answer passed as Spanish; Spanish at 200 is really worse than 3 of 6, and 2 versus 3 means nothing.

The busiest threads

All 15,360 experts ranked by how often the router picks them, from the engine's own running count: 473 million picks, one per expert per word. 90% of that count came with the model files, so this mostly ranks the maker's workload, not this server's. Job labels are rough, rule-based guesses.

The busiest thread of all, layer 4 expert 122, opens and closes code blocks, and fires on 38% of recorded words. No expert takes more than 0.19% of the whole, and none is idle: the quietest still has 20 picks. The top of the list is plumbing, not knowledge: code block edges, maths notation, digits, articles, commas, line breaks. The threads that hold skills sit far down. The five that held Rust together rank between 3,607th and 11,238th, and the Spanish and German teams sit around 5,300th and 8,500th. Busy is not the same as important.

Caveat: our own 82-prompt recording agrees only loosely with this count (correlation 0.57); layer 18 expert 61 is 3rd here but fired only 10 times in 12,600 recorded words.

Reading the threads without pulling them

Everything above was found by running the model and watching. This asks whether the weights alone already say which experts belong to which language. For each of the 15,360 experts, I compared its router weights with the output vectors of each language's words, and gave it the language it lines up with best. No model was run, and the measured language experts were used only to score the guess.

The weights know. Of the 50 Spanish experts found by running the model, 49 sit in the weights' own top 200, and so do 44 of the 50 German ones; only 1 or 2 of the random control experts land there. Across all 1,458 language-tagged experts the weights name the right language 59.5% of the time, against 26.5% for always guessing the commonest and 11% with shuffled tags, and about equally in every layer. English shows nothing, probably because English runs through the whole cloth rather than through threads of its own. Minutes of arithmetic replaced an hour of model runs.

Caveats: the measured tags rest on few prompts, and the word lists for Latin-script languages are short.

Then the proof: I blocked the weights' own picks and graded the answers exactly as before.

Threads found without ever pulling them came apart just like the ones found by watching. Spanish frayed about as fast, its everyday-word share falling from 0.39 to 0.13; German needed about twice the cut. The other language, English, French and Python stayed whole in every run, and the failures had the same texture: languages bleeding together ("dormire porque o sono è un bisogno") and loops ("artisanal y artisanal y artisanal"). Six prompts each, one run, and a grader that knows four languages. Languages may be the easy case, since their words are so distinctive; skills like Rust are the next test.

A skill read from the weights

Languages have their own words, so it's no surprise the weights point at them. A skill like Rust seemed a harder test. I used the same method with 71 Rust-flavoured tokens against random ones, then checked the weights' top 50 against the Rust experts found earlier by blocking.

The five threads that hold Rust together are quiet ones, ranked only 3,607th to 11,238th by use, so counting picks would never find them. The weights point straight at all five. One skill, a hand-picked token list, and the weights' own picks not yet blocked, so this is a lead rather than a proof.

One word, two cloths

Twelve words with two meanings, such as bank (river or money) and crane (bird or machine), each in 5 sentences per meaning: 120 sentences. For each pair of uses I measured how many experts they share, from 0 (none) to 1 (all), averaged over bands of layers.

I went looking for the layer where the two meanings part. There isn't one: they are already apart at layer 4, the earliest layer this recording can be trusted, and stay apart to the end. A river bank and a money bank share about as few threads as two unrelated words. Shuffling the meaning labels makes the gap vanish, so it is real. But the word just before shows much of the same split, so a large part of this is the sentence's subject rather than the word's own meaning. The threads follow the cloth more than the stitch. A test where only one earlier word changes would separate the two.

The threads of thinking

Six prompts answered with thinking on and off, about 7,200 tokens. Which experts work mainly while the model thinks?

Thinking has its own threads, gathered near the very end of the model where words are chosen, and the answer that follows is woven almost the same way whether or not the model thought first. That end position hints these experts write the style of thinking text (hedges, "wait", "let me check") rather than do the reasoning itself. One prompt supplied 39% of the thinking tokens, so this is a first look. Blocking these 610 and grading the answers is the test that would tell.

The two threads that 4-expert mode lets go

The server runs 4 experts per word instead of the model's 6, because it is about 30% faster and tied on the tests so far. This reads the existing recording to see what the 5th and 6th picks were doing. No new model runs.

Dropping two picks lets go of a real third of the weight, yet nothing with its own job disappears with them. The last two threads are thinner copies of the same kinds of thread, which is why short tests cannot tell 4 from 6. It does not prove 4 is safe on long, hard work. Caveats: the weight formula is taken from another engine's code and not yet confirmed inside this one, and about 80% of job labels are still "unclear".

How much can be cut

Least-used experts per layer, blocked through the router on a scratch copy, tested on English, code and security. One run each. Blocking saves no memory; the experts must leave the pack for that, and that part is untested. The set was chosen from 174 prompts, and unusual problems are not tested yet. Use the overlay on the loom to see which experts each level removes.

Where the threads are kept

The graphics card holds 16 GB. After attention, the shared experts and the routers, it has room for about 689 of the 15,360 routed experts; the rest are read from system RAM by the processors. Choose "On the graphics card" in the overlay above to see them on the loom. The engine does not list its real set, so this is my estimate: the 689 most used experts in its saved usage profile. Treat it as rough. Those 689 cover only 20.7% of counted uses in that profile, while the engine's own log shows the card doing 32-46% of expert work on short prompts, so the real set is chosen more cleverly than a simple top list.

Two looms, same cloth

The same model files, run by two different engines, at temperature 0, which should make every run the same. 12 money tasks, each run 4 times, including round-half-up traps like 2.675. Strata splits the work between the graphics card and the processors, and its wording changed from run to run. llama.cpp repeated itself exactly. Speeds include reading the prompt.

With the draft layer off, Strata reached exactly the same final numbers as llama.cpp every time, wrong ones included. Its graphics-card rounding changed the wording, never the answer. The answers that flipped (3928.39 against 3948.39, 2657.64 against 2658.64) came with the draft layer on. The wrong answers are the model's own arithmetic: 38.72 + 25.81 + 12.91 written as 67.44. Repeating a mistake is still a mistake, and DeepSeek made none. One test, 12 tasks; draft on and off also ran with different slot counts.

A thread woven by hand

Everything above is a reading of weights that training wrote. This is the opposite: a network whose every weight was written on purpose. A short integer program that rounds money half up was compiled, step by step, into plain ReLU layers. Each step is a tiny exact gadget: a pair of neurons that turns into a clean 0 or 1 at a threshold, and long division done one bit per layer. Nothing was trained.

For scale, DeepSeek has about 15,360 routed experts of tens of millions of weights each, and still varied from run to run above. These three are 239 to 512 weights and never vary. Changing one weight at a time, the checks caught almost every change. They are slower than plain arithmetic and not connected to any language model; they show that a network's behaviour can be known exactly when it is built, not discovered.

Everything I got wrong along the way

A map you can trust keeps its errors in view.

Changes