Final Integrated Benchmark Report: Qwen3.6-27B-FP8 on Upstream vLLM and Red Hat RHAI vLLM

Executive Summary

All three serving stacks ran Qwen/Qwen3.6-27B-FP8 successfully under the 8K-input, 512-output, concurrency-4 GuideLLM long-reasoning workload. Upstream vLLM 0.21, Red Hat RHAI vLLM 3.4, and upstream latest, which resolved to vLLM 0.22.0 during this test, each completed 12/12 GuideLLM requests without errors.

The recommended Red Hat RHAI vLLM 3.4 customer starting point on this 4 x L4 host remains:

tensor_parallel_size=4
gpu_memory_utilization=0.84
max_model_len=32768
max_num_seqs=64
max_num_batched_tokens=32768

This configuration produced the best observed RHAI throughput in the main GuideLLM run: 55.3 output tokens/s and 927.9 total tokens/s. The gpu_memory_utilization=0.88, max_num_seqs=64, max_num_batched_tokens=32768 variant also passed the main long run and reduced RHAI p95 latency from 76.7s to 70.8s, but with lower output throughput.

The destructive runtime risk is now also measured. RHAI vLLM 3.4 can start successfully and pass health checks, then degrade or fail later under long-context, agent-like concurrency when the scheduling envelope is pushed too far:

• gpu_memory_utilization=0.88, max_num_seqs=128, max_num_batched_tokens=65536 stayed healthy through c4, then at 12K c8 completed 8/16 target requests, produced 8 request errors, health became not_ready, and server logs captured EngineDeadError plus HTTP 500 responses.
• gpu_memory_utilization=0.84, max_num_seqs=256, max_num_batched_tokens=32768 stayed healthy through c4, then at 12K c8 completed 8/16 target requests, health became not_ready, p95 end-to-end latency reached 315.0s, and p95 TPOT reached 724.2ms.
• The 16K-input pilot for 0.88 / 128 / 65536 did not crash, but it became extremely slow at c8: p95 latency 200.6s, p95 TTFT 82018ms, and p95 TPOT 209.0ms while health still returned ready.

Target Host and Runtime

The test target was an AWS EC2-style GPU host. It was a RHEL AI / rpm-ostree-like system, so the test avoided mutable-root assumptions: no changes were made under /usr, and persistent high-write data stayed under /var-backed paths and the NVMe mounts.

Serving Stacks

All serving runs exposed the model through the OpenAI-compatible API and used tensor parallelism across four NVIDIA L4 GPUs.

Main Long-Reasoning Benchmark

Workload: approximately 8K input tokens, 512 output tokens, concurrency 4, 12 GuideLLM requests, 359-second measurement window.

vLLM Built-In Benchmark Reference

The vLLM built-in benchmark was run for upstream vLLM 0.21 as an additional reference point, separate from GuideLLM.

Red Hat RHAI vLLM 3.4 Parameter Search and Recommended Range

The first RHAI 3.4 default-style attempt with max_num_seqs=256 failed during sampler warmup with CUDA OOM. Lowering max_num_seqs was required on this 4 x L4 host before full long-run validation.

Startup and short chat probes passed for these RHAI 3.4 candidates:

Two RHAI parameter sets received full GuideLLM long-run validation:

The 0.84 / 128 / 32768 case passed startup and short chat probing, but it should not replace the recommended setting without full long-run validation at the customer’s target concurrency.

Runtime Degradation Matrix

This matrix tested Red Hat RHAI vLLM 3.4 startup parameters against a fixed long-agent workload shape: approximately 12K input tokens, 640 output tokens, concurrency 1/2/4/8. It is the key evidence for the destructive pattern that appears after successful startup.

The throughput number for failed c8 rows should be read together with success rate and health. For example, 0.88 / 128 / 65536 reports high completed-request throughput at c8 because half the workload failed quickly; the governing signals are 50% success rate, not_ready health, and EngineDeadError.

Longer 16K Degradation Pilot

The 16K-input pilot used gpu_memory_utilization=0.88, max_num_seqs=128, max_num_batched_tokens=65536 with approximately 16K input tokens and 768 output tokens. It demonstrates the slow-response version of the same risk: at c8 the service still answered health checks, but p95 latency and TTFT became customer-visible failures.

Destructive Boundary Summary

Recommendation

Use this RHAI vLLM 3.4 configuration as the customer-facing starting point on this 4 x NVIDIA L4 host:

tensor_parallel_size=4
gpu_memory_utilization=0.84
max_model_len=32768
max_num_seqs=64
max_num_batched_tokens=32768

For latency-sensitive tests, evaluate this variant under the customer’s actual workload:

tensor_parallel_size=4
gpu_memory_utilization=0.88
max_model_len=32768
max_num_seqs=64
max_num_batched_tokens=32768

Avoid treating max_num_seqs, max_num_batched_tokens, and gpu_memory_utilization as independent “more is better” knobs on 4 x L4. This test shows three distinct risk zones:

• Too much startup memory pressure can fail admission before serving starts.
• Larger scheduling envelopes can pass startup but degrade at high long-context concurrency.
• A service can remain health-ready while p95 latency and TTFT are already unacceptable for interactive agent use.

Evidence Package

Final cleanup was verified after the benchmark and degradation work: no vLLM containers remained, no GPU processes were running, and port 8000 had no listener.