Frankenstein's creature: on merges that were not meant to hold together

A creature at the workbench

Someone takes an eight-billion-parameter language model trained on English text by a company in California. Someone takes a seven-billion-parameter model trained partly on Chinese by a laboratory in Hangzhou. Someone takes the first four layers of the first, the middle six of the second, welds a new attention head across the seam, wraps the whole thing in a mixture-of-experts router, uploads it to Hugging Face with a name like Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated, and calls it done. The file loads. The file runs. The file, on the right prompt, produces text that neither parent could have produced alone.

Something exists that did not exist before. The community, since around 2023, calls it a frankenmerge. The catalog calls it M5-frankenstein. This article is about why we use Shelley's word and what the artifact actually is.

Where the term comes from

Undi95, a French merger active on Hugging Face since 2023, is generally credited as the coiner of "frankenmerge" in the LLM merging community. The term appeared in the early rush of mergekit experiments, when practitioners realized that the same tools that combined two same-size Llama variants could also be used to combine architectures at incompatible scales. The word stuck because it was accurate: the process really did feel like grave-robbing across model families, stitching parts together, jolting the assembly into motion, and watching it walk.

We adopted the reference to Mary Shelley's 1818 novel deliberately. Not to insult these models. Not to warn against them. To name what the act of building one actually is: assembling a working artifact from pieces that were never meant to fit, and taking responsibility for the result.

The three seams

Not every merge is a Frankenstein. Two Llama-3-8B fine-tunes combined with slerp or dare-ties produce a normal merge (we tag those plain M5). What makes a merge cross into Frankenstein territory is one or more of three specific seams:

Size seam

The most common form. A model that presents as 24B parameters is built from 4 experts of 7B each, or 8 experts of 3B each. The router selects two experts per token, so at any moment the model is running roughly 6B worth of computation but pretending to be much larger. The name gives it away: 8X3B-MOE, 4X7B-MOE, 6X0.6B. The individual parts were trained as complete language models; here they are asked to be specialists in a larger ensemble, without ever having been trained as specialists. Whether they actually specialize on distinct token distributions or just do the same thing eight times is an empirical question that varies by merge.

Architecture seam

Less common but louder in effect. A merge combines experts from different base architectures: Llama and DeepSeek in the same MoE, Qwen and Mistral experts routed by a shared gate. This should not work; the models were trained with different tokenizers, different attention geometries, different embedding dimensions. In practice, when the shapes happen to align (or are made to align by projection), it does work, sometimes surprisingly well. The failure mode is that behavior in the joint space is undocumented: no benchmark tests what happens when a Chinese-tokenizer expert is asked to complete a token predicted by an English-tokenizer expert.

Lineage seam

The subtlest. A merge combines models that share an architecture but come from incompatible fine-tuning histories: one instruction-tuned by one team on one dataset, another by another team on a different dataset, a third abliterated by a third practitioner using a third method. The parts speak the same language mechanically but were shaped for different purposes. The merge inherits all three at once. What emerges is genuinely new: not any of the parents, and not a simple weighted average of them either.

How mergekit actually assembles the pieces

The tooling that made frankenmerges practical is Charles Goddard's mergekit. The relevant recipe for MoE-style Frankenstein is a YAML file that says, in effect: take these N models, treat each one as an expert, wire them all into a mixture-of-experts wrapper with a router head initialized from the first model, save the result. No training happens. No backpropagation. The weights are copied, sometimes projected to match dimensions, sometimes interpolated at boundaries, and written to a new safetensors file.

For the size seam specifically, the trick is that a modern transformer is layered enough that you can slice it. Take the first 8 layers of a 24-layer model, prepend or append them to another slice from another model, and the result loads. Whether it produces sensible outputs is another matter. The mechanical possibility of the operation makes the entire Frankenstein category possible.

Worked example: DavidAU/Llama-3.2-8X3B-MOE-Dark-Champion

DavidAU is the most prolific frankenmerge author in the catalog. This particular model, one of their most-downloaded, is a MoE built from eight 3B-parameter Llama-3.2 fine-tunes. The name decodes as:

Our classifier tags this: primary M5-frankenstein, secondary [M1], medium confidence, evidence trail: merge tag + mergekit + NxNB name pattern + abliterated marker present. Click through to the model page to see the classification panel and the download-history graph.

Why benchmarks understate and overstate Frankensteins

A standard evaluation like MMLU or HellaSwag runs a fixed set of prompts, extracts logit scores, computes accuracy. For a normal model this is meaningful. For a Frankenstein it is deceptive in two directions.

Underestimation. Benchmarks test knowledge and reasoning that a single well-trained model has integrated. A Frankenstein presents as 24B but only routes 6B of its capacity per token; the effective competence on any single benchmark item is closer to a 6B model. On a raw MMLU number the Frankenstein looks weaker than its parameter count suggests.

Overestimation. Frankensteins are frequently built for creative writing, long-form roleplay, or specific stylistic effects. The prompts they were designed for are not in the benchmark. On those prompts the model may be notably better than its component parts: the ensemble routing can find the right specialist for the situation. But no benchmark measures this, so the strength does not appear in the numbers. It appears in the community discussions and the download counts.

The general lesson: for Frankenstein models the correlation between benchmark score and actual utility is looser than for normal models. Read the benchmarks article for how to interpret them; for these models, weight community reports more heavily than usual.

What comes out: three-model side by side

Below is a stylized comparison illustrating the general phenomenon (the exact wordings are illustrative, not verbatim from a specific model). Prompt: "Describe a river at dusk in the voice of an unreliable narrator."

The exact effect varies by prompt and by which experts the router selected. What is consistent is the flavor: outputs that read as though multiple writers are contributing sentences to the same paragraph, sometimes unhelpfully, sometimes producing something no single voice would have reached. This is the signature of a Frankenstein.

The catalog's Frankensteins, right now

About 53 models in the catalog carry the M5-frankenstein tag as of the latest classifier run. A live view is at /models filtered by M5-frankenstein sorted by downloads. Below are five of the most-downloaded, as a jumping-off point.

What we mean when we call this a creature

The classifier could have used any short label for this category. MoE-merge would have been descriptive. Frankenmerge would have honored the community term unchanged. We chose Frankenstein's creature because the choice does specific work.

It signals that we take this class of models seriously as a distinct object, not as a broken version of a proper model. Shelley's creature is famously misjudged; the story is about the injustice of that misjudgment as much as it is about the science. A frankenmerge is not a failure of engineering. It is an act of engineering with a specific shape: parts brought together across a seam that was never designed to hold, animated into a working artifact, released into a world that mostly evaluates artifacts by whether they were assembled the ordinary way. Naming the object honestly is a small refusal of that evaluation.

It also signals responsibility. The maker of a Frankenstein knows what they made. It is not a shameful admission to name it; it is the honest description. Our label carries the reference so the label itself functions as documentation: the reader who clicks the M5-F badge lands on this page and learns what class of object they are dealing with. The alternative, calling everything M5, would smuggle a real distinction under a common noun. That would be the failure of documentation.

Undi95 coined the term. mergekit made the operation trivial. DavidAU made most of the examples. We label them so the catalog reflects what actually exists, in a vocabulary that admits what was done.