During a bounded comparison, a large coding model collapsed into a degenerate loop on long-form work. The output began cleanly, then repeated a small set of words until the dispatch ended. A different model handled the same prompt shape without that failure. This was an operational observation, not a benchmark study or a general reliability rate.
None of this would have shown up in a public benchmark score. Benchmarks average across many short prompts. The failure mode here only fires on long structured outputs. The benchmark calls the larger model the better model, and that call is accurate for the average case. It is wrong for the use case that breaks it.
What the Failure Looks Like
The degenerate-loop pattern looks like this in practice. The model starts the output on-topic. It produces a paragraph or two of coherent content. Then a word or phrase from earlier in the output gets re-anchored, and from that point forward the model is producing variations on the same fragment for hundreds or thousands of tokens. The output structure (heading markers, bullet markers, section breaks) sometimes continues to look correct, which is what makes the failure invisible to a quick visual scan.
The affected prompts asked for long, structured prose with several sections and consistent formatting. The failed outputs began on topic, then spent most of their remaining length looping over the same fragment.
In that comparison, GLM-4.6 produced usable output on the prompt shape that triggered the loop elsewhere. That does not establish a universal reliability ranking; it only changed the routing choice for this workload.
Why Benchmarks Don’t See This
Public LLM benchmarks score average performance across a corpus of short, well-defined prompts. They report a number. The number is genuinely useful for comparing models on the average case.
Tail-risk failure modes can disappear inside averaged scores when the benchmark does not include the prompt shape that triggers them. A benchmark can rank a model well while missing a failure that matters to a particular production workload.
The shape that triggers this specific failure mode is long structured outputs with multiple sections and consistent formatting requirements. That shape is common in real consulting work, technical writing, and code generation across large codebases. It is rare in benchmark suites because benchmark suites are designed for repeatable scoring, which biases toward short single-section prompts.
The result is that the benchmark numbers and the production reliability numbers disagree by orders of magnitude for some models on some tasks. The model that wins the benchmark can be the model that fails the production workload.
How to Detect It Cheaply
Run the prompt on two backends in parallel and compare the outputs. A large length difference or a mismatch in expected section structure is a reason to inspect the longer output for repetition. These are rough signals, not calibrated thresholds.
For long structured outputs, inspect the tail with a simple frequency analysis. If a small set of words dominates the ending, discard the output and review the routing choice.
Neither check is expensive. Both cost less than reading the full output to discover the failure manually. The cost of NOT checking is shipping a 4,000-token paragraph of “procuring obtaining acquiring” into a document that was supposed to be a competitor teardown.
The Operational Rule That Falls Out
Once a model’s tail-risk failure mode is characterized, the routing rule is straightforward. Use the reliable model as primary for the workload that triggers the failure. Use the larger model as a parallel second opinion when it is worth dispatching, on the assumption that some non-trivial fraction of those dispatches will need to be discarded.
After the comparison, GLM-4.6 became the primary for that long-structured-output workload. The other model remained available as a second opinion, with a tail-pattern check before its output could be used.
What This Generalizes To
The specific failure mode is qwen3-coder:480b in degenerate-loop. The pattern generalizes. Every cloud LLM has tail-risk failure modes that do not appear in its benchmark numbers. The set is different for each model. You will not learn what they are from a leaderboard. You will learn what they are from running real workloads against the model and watching what breaks.
A practical consequence: do not adopt a new cloud LLM as a primary backend until you have run it repeatedly on the shape of work you actually do. Benchmark scores are useful information, but they are not enough.
Parallel dispatch against two models on real workloads is the cheapest way to characterize a new model’s tail risks. The output divergence between the two is your signal.
The Framing
Treat published benchmarks as a hypothesis about the model’s average behavior. Treat your own parallel-dispatch comparisons as the actual measurement of how the model behaves on your work. The two are not the same number. The one that matters for what you ship is the second one.
The model that wins the benchmark and fails the production workload is the most expensive kind of cloud LLM to adopt: the one whose problems do not show up until you have already wired it into your pipeline.