MiMo-V2.6-Pro-RL First Test: vLLM Deployment and Performance Measurement on 8x H20-3e¶

Getting to Know the Pro Model¶

MiMo-V2.6-Pro-RL is the flagship open-weight model of Xiaomi's MiMo-V2.6 series, using a sparse mixture-of-experts architecture with approximately 1.02T total parameters and about 42B activated parameters per token. The official model card lists text, image, video, and audio inputs and a 1M-token context capability, under the MIT license. Compared with the Flash version, Pro's weights and per-token computation are significantly larger, so deployment first faces weight loading, expert routing, and large GPU-memory communication.

The configuration used in this round has 70 layers, 384 routed experts, and Top-8 expert selection; the main weights include FP8 and MXFP4 expert layouts. The official release also provides multi-token prediction related weights, but this round's baseline did not enable speculative decoding. The 1M context listed on the model card is a model capability; this round's serving window is fixed at 32,768 tokens, and the two should not be equated.

The model replica has 129 main weight shards, totaling approximately 563.58 GB, plus MTP weights. The experiment reused a model replica that had already been synchronized and verified; before startup, we rechecked the main shard count, total byte size, configuration, and temporary file status, without repeating a full weight hash.

Hardware and Test Methodology¶

The experiment used a single node with 8×H20-3e, vLLM tensor parallelism TP8, and the model was loaded from shared read-only storage. Before formal stress testing, we completed 8-rank communication checks, model ID, simple arithmetic, multi-turn memory, streaming termination, and usage validation. The model service uses an OpenAI-compatible interface, with Prefix Cache disabled, and sets gpu-memory-utilization=0.90 and a 32K serving window. The stress-test input is constructed by repeating a fixed English technical text, trimmed to the target token length, with temperature=0, min_tokens=max_tokens, and ignore_eos=true set; this measures the inference system's capacity under this synthetic load, not the model's answer quality.

Each scenario runs at three concurrency levels: C1, C8, and C32. After warm-up for each level, three formal rounds are repeated. C1 has 8 requests per round, C8 has 16 requests per round, and C32 has 64 requests per round. A valid request must have a successful response, non-empty output, a complete termination marker, valid usage, and the target output length; failures remain in the original denominator. The results in the table are the median of three formal rounds. TTFT is the wait time for the first non-empty token, TPOT is the average time per output token after the first token, and E2E is the total request duration.

vLLM 8-Card Baseline¶

vLLM completed 36/36 formal test rounds, with 1056/1056 requests successful. Model ID, arithmetic, multi-turn, streaming integrity, tool calls, and tool result feedback all passed; image validation covered two different images and the answer changes after swapping their order. Passing images does not equal passing video or audio: the local model replica does not yet have complete audio tokenizer and DFlash files, so this round does not report audio or speculative decoding results based on that.

This vLLM service is also registered in the existing OpenWebUI. Sending 'Calculate 17 plus 28, answer only the number' from the interface returns 45. This check covers the complete text request path from browser, OpenWebUI, cross-cluster ingress, to the inference service.

The actual prompt examples used in functional acceptance are as follows. The image tests use carrot and corn images, asking the model to answer only the English name of the main vegetable, then swap the order of the two images to verify the answer order.

For short requests, increasing from C1 to C32 raises total output throughput by about 5.3 times, but P95 TTFT grows from 0.20 seconds to 4.20 seconds. At 24K input with C32, although all requests complete, P95 time to first token approaches 96 seconds; online services should decide admission concurrency based on latency targets rather than only final success rate.

Within the formal stress-test time window, the utilization peak of all eight cards was 100%, GPU memory usage peaked at about 131 GB per card, power consumption peaked at about 364–396 W, and temperature peaked at about 52–65°C. These are discretely sampled curves and peaks, not precise energy consumption under continuous sampling.

Deployment Recommendations¶

Eight cards are the verified vLLM starting point for this round. Before going live, check the GPU topology and NCCL communication actually obtained by the container, especially cross-GPU activation and expert communication under 8-card tensor parallelism. CPU-side tokenization, vision preprocessing, and host-to-GPU-memory data copies are also affected by NUMA locality; it is recommended to include CPU, memory, and GPU affinity in acceptance.

Capacity planning should be layered by request length: the latency curves for 1K conversation, 8K retrieval-augmented, and 24K long-document differ significantly. Production traffic should set separate concurrency and queue limits, and simultaneously monitor success rate, TTFT, TPOT, E2E, GPU utilization, and GPU memory. Prefix Cache, speculative decoding, and contexts longer than 32K each require establishing new baselines and should not directly reuse the table above.

References¶

• Xiaomi MiMo-V2.6-Pro-RL official model card