Quantization, KV-Cache Compression, and Real Coding Benchmarks

The earlier cluster benchmark write-up asked which frontier models were fastest. This post asks the next question: once hardware, serving stack, and benchmark shape are held constant, what do deployment choices - weight quantization and KV-cache compression - actually do to real coding-agent behavior? The model family stays fixed; the things operators actually touch get varied.

This grew out of a r/LocalLLaMA thread asking for real coding benchmarks across quantization levels rather than proxy metrics. The study plan: Qwen/Qwen3.6-27B as the primary sweep, Qwen3-Coder-30B-A3B-Instruct as the coding-specialist cross-check, and MiniMaxAI/MiniMax-M2.1 as the heavyweight extension. That last lane turned out to be the most instructive result of the study, and not in the way I expected.

TL;DR

Official FP8 beat BF16 on both benchmark score and every operational metric - not the “safe compromise” result I expected. Community AWQ stayed competitive on coding quality and edged ahead on pass@2, but was slower. A W4A16/GPTQ checkpoint loaded fine and returned punctuation-only garbage, which is recorded as a real incompatibility result. Calibrated fp8 KV recovered strong results; uncalibrated did not. MiniMax-M2.1 is a clear negative: the setup bug was fixed, and the model still collapsed under the edit/test loop. The local SWE-bench Pro path now works end-to-end, but the Qwen canaries are path-proving runs, not headline scores.

At a Glance

Each row below is a “lane” - a specific combination of a model checkpoint, a runtime container, and a serving configuration. Keeping them distinct is the whole point of the study; the full methodology is in Section 2.

pass@1 = solved on a single attempt. pass@2 = solved on at least one of two independent attempts. A higher pass@2 with a large gap relative to pass@1 means the model can solve a problem but does so inconsistently.

1. Hardware and Study Scope

Same cluster as the earlier write-up: 16 x H200 across two Dell PowerEdge XE9680L nodes, each with 8 x NVIDIA H200, dual Intel Xeon Platinum 8570, 2.0 TiB of system RAM, and 8 x 3.84 TB of local NVMe - roughly 2.30 TB of aggregate HBM, 4.0 TiB of host RAM, and a multi-rail InfiniBand fabric. This time it is being used as a controlled environment for a narrower question, not a broad model bake-off.

The model set is:

• Qwen/Qwen3.6-27B
• Qwen/Qwen3-Coder-30B-A3B-Instruct
• MiniMaxAI/MiniMax-M2.1

1.1 Why these three?

• Qwen/Qwen3.6-27B - practical enough to sweep thoroughly, strong enough on coding tasks to matter, and directly aligned with the original Reddit ask.
• Qwen3-Coder-30B-A3B-Instruct - the coding-specialist cross-check: does the quantization story from a general-purpose model hold up on one explicitly positioned for coding?
• MiniMax-M2.1 - the heavyweight extension: what does the bigger cluster let you try once the Qwen sweeps are understood? That lane ended up producing the clearest lesson of the study.

What Was Left Out, And Why

• SLMs were out of scope - wrong size range for this question. Whether FP8 calibration matters on a 16-node H200 cluster is not a question you answer using consumer GPU models.
• Very large Part 1 models (Qwen3-235B, Kimi-K2.6, Mistral Large 3, Llama 4 Scout, GLM-5.1-FP8) were out for the opposite reason: too expensive to sweep multiple variants. You cannot cleanly iterate over six checkpoints when each run costs an order of magnitude more compute.
• DeepSeek-V4-Pro was blocked by a fused-router dtype bug in vLLM when this sweep ran ( #40862 , since fixed in #43425 ), making it a poor target for a controlled comparison at the time. DeepSeek-V4-Flash was available but not chosen; Qwen gave a cleaner testbed for the quantization question.
• Random community low-bit checkpoints were excluded unless widely used or with a documented vLLM path - otherwise this becomes checkpoint archaeology. The W4A16/GPTQ lane made the cut because it was documented and widely shared, and its result still taught something real.

What is a Lane?

Throughout this post, “lane” refers to a specific combination of a model checkpoint, a runtime container, and a serving configuration. Keeping that combination explicit matters because two checkpoints that look similar on paper can behave very differently when one of those variables changes, and the whole point of this study is to isolate which variable is responsible for what.

2. Method

The rule from the earlier benchmark work: hold the benchmark shape fixed before changing another variable. Two benchmarks were used.

Aider Polyglot is the fast iteration lane. Aider acts as a coding agent - it reads code, proposes edits, runs the test suite, and retries until the tests pass, the same loop a developer uses. The Polyglot benchmark runs that loop against a fixed set of programming exercises across multiple languages. The lane here was 34 Python cases, all with the same harness, prompt shape, and serving stack. It runs fast enough to sweep multiple checkpoint variants in a single day, and it tests something single-turn benchmarks miss: whether a model stays coherent across tool calls, test failures, and patch revisions.

SWE-bench Pro is the serious lane. The model receives a real GitHub issue filed against a real open-source project and must produce a patch to resolve it. It is slower and more expensive per case, so it was reserved for the Qwen closeout run rather than being used for every quant variant. The Pro variant matters: as of early 2026, the older Verified has too much data contamination to be a reliable signal for frontier-model comparisons.

A note on the metrics

• pass@1 is the fraction of problems the model solved correctly on a single attempt.
• pass@2 is the fraction it solved on at least one of two independent attempts.

Both are reported throughout. A large gap between pass@2 and pass@1 means the model can solve the problem but not reliably - it might succeed on the second try after failing the first. A tight gap means it is either consistently right or consistently wrong. For a coding agent, that consistency matters as much as the peak score.

For inference: vLLM, serving a standard OpenAI-compatible API that the benchmark harness calls. Almost everything ran on vllm/vllm-openai:v0.19.1 - all Qwen sweeps and the Qwen3-Coder lane. Pinning one container version across those runs is the point: score differences are then attributable to the checkpoint and serving config, not an uncontrolled runtime change.

The one exception was MiniMax, which required its own official build: vllm/vllm-openai:minimax27. MiniMax’s architecture requires runtime support that is not yet available in the standard release, so their team maintains a dedicated container. The rule is the same as in the earlier work: use the standard lane unless the model genuinely requires otherwise.

2.2 Validation and Artifact Capture

Beyond the score, every run captured the full artifact trail: benchmark stdout and stats, server logs, engine-throughput logs, node telemetry, smoke outputs, and normalized artifact manifests. When you vary one parameter at a time, the differences are often subtle enough that you need that trial to understand why two similar-looking runs behaved differently. Langfuse was also wired in as a cross-check, so the bar for a “complete” run was not merely “the server came up.” A run needed to complete, produce artifacts, and emit the expected trace metadata. A failing run was still informative rather than just a gap in the table.

2.3 Representative Commands

A few sanitized command examples are useful here to make the setup concrete - not a full pipeline write-up, but enough to show the shape of what ran.

The vLLM serve lane for Qwen/Qwen3.6-27B-FP8 - the baseline against which all quantization and KV-cache comparisons are measured:

docker run -d \
  --name qwen36-27b-fp8 \
  --gpus all \
  --ipc=host \
  --network=host \
  --shm-size=32g \
  --device /dev/infiniband \
  -v /home/amit/vllm-cache/hf:/root/.cache/huggingface \
  -e HF_HUB_ENABLE_HF_TRANSFER=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=7200 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  vllm/vllm-openai:v0.19.1 \
  Qwen/Qwen3.6-27B-FP8 \
  --served-model-name Qwen/Qwen3.6-27B-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 131072 \
  --tensor-parallel-size 8 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --language-model-only \
  --trust-remote-code

The fixed-harness Aider Polyglot invocation - the model changes between lanes, everything else stays constant:

docker run --rm \
  --memory=12g \
  --memory-swap=12g \
  --add-host=host.docker.internal:host-gateway \
  -w /aider \
  -v /home/amit/vllm-part2/.bench/src/aider:/aider \
  -v /home/amit/vllm-part2/.bench/src/polyglot-benchmark:/polyglot-benchmark \
  -v /home/amit/vllm-part2:/part2 \
  -e OPENAI_API_KEY=EMPTY \
  -e OPENAI_API_BASE=http://<cluster-node>:8000/v1 \
  aider-benchmark \
  python3 benchmark/benchmark.py qwen36-27b-fp8-full \
    --new \
    --model openai/Qwen/Qwen3.6-27B-FP8 \
    --edit-format diff \
    --threads 1 \
    --read-model-settings /part2/configs/aider/qwen36_27b_local_nothink.yml \
    --exercises-dir /polyglot-benchmark \
    --languages python

For the KV-cache sweep, the only difference was a single runtime flag - which is part of why the result is interesting:

# baseline
--kv-cache-dtype auto --no-calculate-kv-scales

# uncalibrated fp8 KV
--kv-cache-dtype fp8 --no-calculate-kv-scales

# calibrated fp8 KV
--kv-cache-dtype fp8 --calculate-kv-scales

3. What Is Being Compared?

Two distinct mechanics - not the same operation at different stages.

Weight Quantization

Quantization changes how model weights are stored and executed. Think of it like audio compression: 128 kbps saves space and may be indistinguishable for casual listening, but a professional producer will hear exactly where it falls apart. Quantized weights can look fine on synthetic benchmarks while quietly degrading on extended reasoning and multi-turn tool use - the kind of pressure a coding agent actually applies. The only way to know is to run the real tasks.

Weight Formats

BF16 (bfloat16) is the 16-bit floating-point format used for full-fidelity inference; it is the baseline, carrying the full weight resolution the model was trained with. FP8 is an 8-bit floating-point format natively supported on H100 and H200 hardware; Qwen publishes official FP8 checkpoints that are specifically calibrated to minimize quality loss, rather than simply truncating the full-precision weights. Halving the bit-width roughly halves the memory the model occupies in GPU memory, and the H200’s native FP8 execution units can run it faster than BF16.

Community Low-Bit Checkpoints

AWQ (Activation-Aware Weight Quantization) goes further: it analyzes actual activation patterns during a calibration pass to identify which weights are most sensitive, then compresses the less sensitive ones more aggressively, rather than treating all weights equally. The result is a typically 4-bit checkpoint that degrades less than naive 4-bit quantization would suggest.

W4A16/GPTQ is a different approach: weights are stored as 4-bit integers (W4) while activations stay at 16-bit float precision during computation (A16), using the GPTQ algorithm to find the per-layer 4-bit representation that minimizes reconstruction error. It is a more aggressive compression target than AWQ, and as the results here show, that aggressiveness has a real operational cost at certain serving configurations.

KV-Cache Compression

The second mechanic is KV-cache compression. This is a different axis entirely. The checkpoint stays fixed; what changes is how the runtime stores the key-value activations during generation. The three modes in this study are: baseline KV cache, fp8 KV without calibration, and fp8 KV with calibrated scale calculation. The theoretical appeal of compressed KV is that it reduces memory pressure during long decodes, which matters greatly for a coding agent that can run for many turns. The practical risk is that precision loss compounds across long sequences in ways that are hard to predict from short-context behavior alone, and the difference between calibrated and uncalibrated compression turned out to matter quite a bit.

4. Starting Hypotheses

Before running anything, my going-in hypothesis was that the official FP8 would be the clear winner: half the memory footprint of BF16, native hardware support on H200, and Qwen’s own calibrated checkpoint rather than a third-party conversion. That combination felt hard to beat for day-to-day use.

For the more aggressive low-bit checkpoints - AWQ and W4A16/GPTQ - I expected poor quality and the thing to start breaking down; maybe not catastrophically, but enough to show up under the edit/test/retry loop pressure of a coding agent, where a model has to stay coherent across multiple rounds rather than answer once.

And finally, for KV-cache compression, I expected smaller and harder-to-read effects. The theory is appealing - compressed KV means less memory pressure during long decodes - but the risk of precision loss compounding over many turns felt real, so I expected the result to be noisy rather than clean.

Most of that was wrong, or at least incomplete. Let’s dig in and see what happened. 🤓

5. Qwen 3.6 27B: The Primary Study

I ran four versions of Qwen/Qwen3.6-27B through the benchmark: the official BF16, the official FP8, the community AWQ 4-bit, and the third-party W4A16/GPTQ 4-bit. Three produced scores. The fourth booted cleanly, accepted requests, and returned punctuation-only garbage for every prompt - worth recording rather than skipping, because “it loads but doesn’t work” is a useful data point for anyone evaluating that checkpoint.

5.1 First Real Result: BF16 vs Official FP8

The first completed lane was a controlled first cut on Qwen/Qwen3.6-27B across two official weight formats:

• Qwen/Qwen3.6-27B in BF16
• Qwen/Qwen3.6-27B-FP8

The runtime, benchmark, and prompt shape were held constant, with the results on the Python Aider Polyglot lane below.

FP8 did not just match BF16 - it beat it on every axis. Decode throughput was 190-192 tok/s versus 170-175 tok/s. Container memory was 40.4 GiB versus 43.3 GiB. GPU power was a steady 240-245W versus a wide, hot 287-330W band. And the benchmark score wasn’t even close.

That changes the default assumption for this model line. The usual framing for FP8 is “a safe compromise that mostly preserves quality.” On this benchmark shape, it was not a compromise - it was the stronger format. Official FP8 is the practical reference point, not a side experiment.

5.2 Second Result: AWQ Changes The Story Again

The next completed lane was QuantTrio/Qwen3.6-27B-AWQ on the same Python Aider Polyglot slice.

This result did not behave the way a simple “lower bits mean worse coding quality” story would predict.

Operationally, AWQ was clearly less pleasant than official FP8. Decode sat around 154-155 tok/s, noticeably slower than the FP8 lane. Container memory at roughly 40.9 GiB was comparable, but bring-up took materially longer because graph capture and warmup were significantly slower. Power draw sat in a 216-235W band, lower than both BF16 and FP8, but at the cost of slower generation.

What surprised AWQ was the coding metric itself. The pass@2 ordering ended up as follows: AWQ at 88.2%, official FP8 at 85.3%, BF16 at 70.6%.

Official FP8 is the cleanest operational reference - faster startup, faster decode, better on every system metric. But AWQ shows a more aggressive community quant can stay competitive on coding quality and edge ahead on pass@2 if you accept the slower bring-up and per-case runtime. That is an honest trade-off, not a pure quality regression.

5.3 Third Result: A Low-Bit Checkpoint Can Still Fail In Practice

The next attempted lane was LibertAIDAI/Qwen3.6-27B-W4A16-G128, a third-party GPTQ/W4A16 checkpoint with a documented vLLM 0.19.1 path.

The lane did not fail because vLLM could not load it. It loaded, warmed up, and served requests. The failure was subtler and more important: under our TP=8 serving shape (Tensor Parallelism degree 8, meaning the model weights are sharded across all 8 H200s on a single node) the endpoint returned punctuation-only garbage even for trivial prompts. The same behavior showed up in the benchmark lane and in direct endpoint probes, so this was not a benchmark-harness bug; it was a bad deployment result for this exact serving configuration.

So W4A16/GPTQ does not enter the scored table for this study. It is recorded instead as an incompatibility or instability result for this cluster shape.

That is still a real result. A checkpoint that boots but answers 2+2 with repeated punctuation is not a usable coding deployment, no matter how attractive the disk footprint looks on paper.

5.4 The Operator Details That Actually Mattered

The score table only tells part of the story. Here are the deployment details that actually changed the practical read:

The footprint differences were real, but disk size alone did not predict deployment behavior. FP8 and AWQ were both much smaller than BF16, yet their warmup and graph-capture behavior still diverged enough to matter. The useful question turned out to be less “which checkpoint is smallest?” and more “which is small enough while still behaving predictably under a real coding-agent loop?”

6. KV-Cache Sweep: FP8 KV Is Not One Knob

After the weight-format sweep, I held the model fixed at official Qwen/Qwen3.6-27B-FP8 and varied only KV-cache mode:

• baseline kvauto
• fp8 KV without calibration
• fp8 KV with scale calculation enabled

Results on the same Python Aider Polyglot lane:

The calibration step matters because KV-cache compression introduces rounding error into stored key-value activations. Without it, quantization scales are chosen from a generic prior about activation magnitude. With it, they are derived from actual representative inputs. That difference shows up directly in the table: uncalibrated fp8 KV introduced a malformed output and an error case; calibrated’ fp8’ KV recovered the clean 85.3% pass@2 result. Uncalibrated is not a free optimization. If I carry compressed KV forward on this model, it has to be the calibrated path.

7. The Serious Lane: SWE-bench Pro Now Runs Locally

What I needed from SWE-bench Pro at this stage was not a leaderboard claim - just a clean answer on whether the local self-hosted path actually works end-to-end.

It does. Two Qwen/Qwen3.6-27B-FP8 canary runs completed locally and produced prediction artifacts. The tighter rerun booted the benchmark image, reset the repo, uploaded tools, entered the agent loop, produced a patch, and autosubmitted a prediction.

The blocker is no longer “can this benchmark run locally?” It is now what the benchmark is actually supposed to measure: model quality inside the agent loop. A path-proving canary and a publishable score are different things.

To summarize where things stand: official FP8 is the strongest weight format. AWQ is the best community low-bit result. Calibrated fp8 KV is the only compressed-KV path I would carry forward. W4A16/GPTQ is a real incompatibility for this TP=8 shape. And the local SWE-bench Pro path now works, but the canary runs are proof-of-path, not leaderboard scores.

8. Extension Lane: Qwen3-Coder 30B A3B Instruct

Qwen/Qwen3-Coder-30B-A3B-Instruct was the cross-check: does the same deployment story hold up on a model explicitly aimed at coding, or was the Qwen3.6 result a quirk of one general-purpose line?

The first 3-case canary looked promising: pass@1 33.3%, pass@2 66.7%, 100% well-formed outputs, zero malformed responses, zero error outputs, zero exhausted context windows, and 5.3s/case.

Before the full run, I patched the local Aider metadata so the model got an 8192 history budget instead of the default 1024. That avoided the same unfair setup problem that later became clearer with MiniMax.

On the full Python lane, the result settled lower:

That is not a great headline score, but it is still a useful result.

Operationally, Qwen3-Coder was well-behaved: it stayed inside the edit/test workflow, no context exhaustion, few malformed outputs, generation bursts around 240 tok/s. A modest headline score and a stable, predictable agent loop are not mutually exclusive.

Qwen3-Coder did not beat the best Qwen3.6-27B lanes on score, but it reinforced something more useful: a model’s behavior inside the agent loop matters at least as much as its benchmark positioning.

9. Extension Lane: MiniMax Was Worth Testing, But Not Worth Blindly Pushing

MiniMaxAI/MiniMax-M2.1 was the heavyweight extension - the obvious “use the big cluster for something unusual” lane. The question was whether raw scale rescued coding-agent behavior.

The first MiniMax canary was not fair because Aider was only giving the model a 1024 token history budget. I corrected that with local model metadata and reran it with an 8192 history budget.

That corrected canary improved to: pass@1 0.0%, pass@2 66.7%, 100% well-formed outputs, 4 error outputs, 4 exhausted context windows, and 54.9s/case.

That made the next decision simple: run one real full lane and see whether the model stabilized.

It did not.

MiniMax full Python lane:

MiniMax did not fail because it could not serve. The runtime was fine. The cluster was fine. The first harness bug was fixed. What failed was the model’s fit for this exact coding-agent workflow. Inside Aider, MiniMax remained too verbose, consumed too much context, and progressively degraded under the tight edit/test/retry loop. The 63 error outputs and 43 exhausted context windows are not noise; they are a description of a model that, in this harness, could not remain disciplined within the token budget required by the agent workflow.

This is not a verdict on MiniMax as a model. Some models are built for long-context repository reasoning rather than repeated short-action tool use, and a different harness shape might tell a different story. But in this study, the setup bug was real, it was fixed, and the model still failed badly. That makes it a meaningful negative result, not a missing datapoint.

10. A Real Blocker: Getting SWE-bench Pro To Run Locally

The serious benchmark lane was not a “pip install and go” experience. The first blocker was the benchmark environment, not the model.

The local SWE-bench Pro canary initially failed before the first real task step for a few different reasons:

• The generated Pro task images came with their own entrypoint assumptions, which conflicted with the way SWE-agent and SWE-ReX expected to launch the runtime
• The default local Docker path in SWE-ReX tried to inject its own Python bootstrap assumptions that did not match those of the benchmark images
• Once that was bypassed, the fallback install path ran into Debian PEP 668 protections inside the task container

The recovery path was:

• switch the generated Pro instances from the simple file format to explicit expert_file instances
• Clear the benchmark image entry point so the runtime launch command was actually the command being executed
• patch the local SWE-ReX fallback path so the ephemeral container could bootstrap swerex-remote successfully under this image layout
• tighten the local Qwen canary prompt so the model would take smaller, more tool-shaped actions instead of dumping giant multi-block plans

Two practical takeaways: benchmark setup is part of the benchmark, and the self-hosted path is workable but not yet friction-free. If you are trying to reproduce SWE-bench Pro locally with open models, expect to do some plumbing.

11. How Agents Were Used In This Study

There are two distinct agent roles in this work, and they are worth separating.

The benchmark subject is the model being tested inside Aider Polyglot or SWE-agent - the thing being measured.

The operational agent is Codex, used to orchestrate the experiment itself: building and revising benchmark runners, launching serving lanes, collecting artifacts and telemetry, updating the draft as results landed, and debugging harness failures (including the SWE-bench Pro path) without losing state on the rest of the study. When W4A16/GPTQ returned punctuation garbage, the useful outcome was not to retry - it was to prove the result, record it as an incompatibility, and move on. When SWE-bench Pro failed before the first model step, the agent traced the failure from image entrypoints through SWE-ReX bootstrap assumptions to Debian package policy and turned the fixes into reusable scripts. As each lane finished, artifact bundles, notes, and branch pushes happened as part of the run rather than being deferred to the end.

That split matters more than it might look. Running a benchmark and orchestrating a benchmark study are genuinely different jobs, and agent tooling is increasingly useful for the second one.

12. Summary

On weight quantization: official FP8 on Qwen/Qwen3.6-27B is a strong default reference, not just a compromise format. AWQ can stay genuinely competitive on coding-agent tasks even when it is slower and operationally messier. And some low-bit formats should be treated as deployment-risky until proven under the exact cluster shape you care about: the W4A16/GPTQ result is a good reminder that a checkpoint that loads without errors can still be functionally broken in a specific serving configuration.

On KV-cache compression: calibrated fp8 KV can preserve coding-agent quality much better than an uncalibrated path. The uncalibrated version is not a free optimization; the calibration step matters more than you might expect from reading the implementation notes.

On model selection: a coding-tuned model is not automatically the winner if its behavior in the agent loop is weaker than that of a more general model. And a larger model is not automatically better if it burns context and fails to stay disciplined inside the harness. MiniMax-M2.1 is the clearest example of both of those lessons landing at once.

The question was never “which quantization format saves the most memory.” It was which formats, runtime settings, and model families actually hold up when you put them inside a real coding-agent workflow. That is what this study measured.

12.1 Hypotheses vs. Reality

Worth closing the loop on Section 4, because the hypotheses did not all survive.

What held up:

Official FP8 really was a strong practical default on Qwen/Qwen3.6-27B, stronger, in fact, than the “safe compromise” framing usually implies. KV-cache compression was really more implementation-sensitive than the weight-format story: the same fp8 KV mode behaved very differently when calibrated versus uncalibrated, which would have been invisible if only one variant had been measured. And the calibrated FP8 KV was clearly the more credible of the two paths.

What got disproved, or at least seriously weakened:

• BF16 is not automatically the safest choice - FP8 beat it on every dimension.
• More aggressive low-bit quantization does not automatically hurt coding quality - AWQ stayed competitive and edged ahead on pass@2.
• A coding-tuned model does not automatically win - Qwen3-Coder was cleaner operationally but did not beat the best Qwen3.6-27B lanes.
• A bigger model does not behave better as an agent - MiniMax disproved that clearly.
• A checkpoint loading successfully does not mean it is a valid deployment - W4A16/GPTQ loaded fine and returned punctuation noise.

The shortest version of all of that: operationally clean FP8 can be a genuine win, not just a compromise. And bigger, more “coding-specialized,” or “lower-bit-by-default” does not reliably predict better coding-agent outcomes. The results have to come from running the real tasks.