This benchmark compares an unquantized Gemma 2 9B model with an FP8 quantized variant on an NVIDIA L4 GPU, revealing that FP8 quantization introduces a prefill tax (higher TTFT) but improves decoding latency and VRAM usage, with minimal semantic drift for narrow tasks.
When evaluating migrating production LLM workloads off commercial cloud APIs, the conversation usually gets oversimplified into a trade-off between quality and infrastructure cost. To look past clean, isolated averages, I built a repeatable evaluation matrix using a real-world workload: cold outreach and contextual profile re-engineering for my resume generation platform. I benchmarked an unquantized Gemma 2 9B against an optimized FP8 variant served via vLLM on a single commodity NVIDIA L4 GPU. The dataset evaluates dynamic text generation across diverse recipient personas, varied complexity buckets (short to long contexts), and strict integer formality parameters. I captured client-side and server-side telemetry to audit how FP8 compression changes runtime reality. The base evaluation set is public at rsher60/resume-gen-benchmark. Here is the raw telemetry and the infrastructure trade-offs I uncovered. 1. Time to First Token (TTFT): The Hidden Prefill Tax of Quantization The dominant open-source narrative is that FP8 quantization makes everything faster. However, if your application is highly interactive and streaming to a UI, TTFT is the only metric that dictates perceived user speed. My telemetry exposed a classic hardware-software trade-off: The Prefill Penalty: For complex, long-context prompts targeted at high-complexity personas, the unquantized model returned tokens to the server in 866.93ms. The FP8 variant spiked to 1372.12ms—a 58% latency penalty on the initial prefill. Why this happens: Quantization reduces memory bandwidth bottlenecks during token generation (the decoding phase). However, the matrix-multiplication de-quantization overhead during the heavy, compute-bound prefill phase introduces a noticeable tax on long input tokens when running on compute-bound commodity hardware like the L4. Production Edge Cases: I caught a massive TTFT spike on the FP8 model during short-context runs, hitting 1,740.34ms. This reflects live infrastructure realities under vLLM scheduling—such as a cold prefill or context block swapping. It proves you cannot evaluate architecture purely on clean, isolated averages. 2. End-to-End Latency: Where FP8 Wins the Generation War While FP8 forces you to pay a tax on the prefill, it aggressively earns its keep during the steady-state decoding loop where the LLM is heavily memory-bandwidth bound. By dropping the weight precision down to 8-bit integers, the amount of data moving across the GPU memory bus is sliced roughly in half. For medium-length generation sequences, the average client total time dropped from 12,290.2ms to 11,526.2ms. If your application handles medium-to-short context sizes or runs entirely asynchronous/batch tasks, FP8 provides a clean, deterministic infrastructure win. 3. The Quality Ledger: Did 9B Parameters Hold the Line? I verified the generated outputs of the raw unquantized runs against the FP8 model outputs (rsher60/resume-gen-benchmark-results). Schema & Persona Adherence: For targeted, single-turn tasks like tailoring text based on a fixed personal profile, a carefully designed system prompt ensures that the 9B architecture executes with near-identical formatting and persona fidelity as a frontier model. Semantic Drift: For narrow, domain-specific tasks, FP8 quantization introduced practically negligible semantic drift. The model successfully retained complex context keys—matching the tone for a cold outreach to an engineer versus a formal application letter—while executing within a significantly lower memory footprint. Strategic Architectural Takeaways Interactive/Low-Batching/Long Inputs: Unquantized weights or a highly aggressive, unchunked prefill strategy might be required to protect your TTFT and prevent user UI friction. Asynchronous/Streaming/Short-to-Medium Context: FP8 is an absolute necessity. The real reason to run FP8 on an L4 isn't just saving a few hundred milliseconds of total latency—it’s the VRAM liberation. Shrinking the model footprint frees up massive amounts of memory for the KV Cache, allowing you to scale concurrency without throwing Out-Of-Memory (OOM) exceptions. I put together the complete analysis, including the upcoming vLLM configurations and cache allocation strategies I used to sustain 92.7% KV Cache utilization under heavy concurrent load, in the full write-up here: https://billionars.substack.com/p/benchmarking-my-self-hosted-gemma HF datasets here: https://huggingface.co/datasets/rsher60/resume-gen-benchmark https://huggingface.co/datasets/rsher60/resume-gen-benchmark-results https://huggingface.co/datasets/rsher60/resume-gen-benchmark-results-optimised
A developer created a task-aware GGUF quantization pipeline that uses tensor-level bit allocation to improve Gemma 4 12B Q3 coding performance by 8.55% over a hand-tuned imatrix while increasing model size by only 0.119%.
Unsloth has released KL Divergence benchmarks for Gemma 4 26B-A4B GGUF quantizations, showing Unsloth GGUFs top 21 of 22 sizes on the Pareto frontier. They also introduced a new UD-IQ4_NL_XL quant fitting in 16GB VRAM and updated Q6_K and MLX quants for both Gemma 4 and Qwen3.6.
Gemma 4 12B QAT (dense) achieves over 1000 tokens per second prefill on an 8GB RTX 4060 with 120k context using TurboQuant, enabling full GPU layer offloading. This represents a 42% increase in prefill speed over previous methods.
A user compares Qwen3.6 35B-A3B and Gemma 4 26B-A4B-IT running locally on a 16GB VRAM GPU via LM Studio, finding Qwen3.6 produces more detailed outputs while both run at comparable speeds. The post is an informal community comparison using quantized models.
A user benchmarks Google's Gemma 4 QAT models on an AMD 7900 XTX, reporting up to 45% faster generation, 83% higher throughput, and significant VRAM savings (e.g., 5.7GB for the 12B QAT model) with no quality loss compared to standard weights.