中/EN
Field SOP
Field SOP

Can One GPU Run Matrix-Game 4.0? A Local Deploy SOP

A nine-step SOP for running Matrix-Game 4.0 locally on a single GPU, with every command and flag taken verbatim from the v4 subdirectory README (repository: github.com/SkyworkAI/Matrix-Game, file Matrix-Game-4/README.md). Step 1, align on requirements: model inference needs Linux, an NVIDIA GPU and Python 3.12 together, the reference A100 80 GB deployment uses PyTorch 2.12.0 / CUDA 13.2, and the repo ships a dedicated RTX 5090 runtime that does not auto-detect hardware, so the matching preset must be selected explicitly; the browser demo needs a Linux server that also builds the game and GStreamer pipeline while the Safari/Chrome client needs no GPU; standalone play and recording need macOS or Linux plus CMake, C++20, SDL2 and FFmpeg, with the legacy Windows entry retained but not validated for feature parity; and weights and assets are not shipped in the repo, so budget extra space for the Hugging Face Hub cache. Step 2, install inside the Matrix-Game-4/ subdirectory or everything after is wrong: git clone, cd Matrix-Game/Matrix-Game-4, python3.12 -m venv .venv and source .venv/bin/activate, then pip install torch==2.12.0 torchvision==0.27.0 --index-url https://download.pytorch.org/whl/cu132 as a matched pair, then pip install -e '.[inference,server,download,test]' with all four extras; FlashAttention is optional and only rtx5090-sage needs SageAttention while plain rtx5090 uses PyTorch SDPA. Step 3, downloads come as two manifests, game assets and model weights, pinned to revision 4419accdb7bb7881234c5914de8cd915266e0325 with --scene switchable across castle, racing and all, and one server process supporting only one scene model; the downloader verifies size and SHA256 against the manifests so interrupted runs can simply restart, shared T5/tokenizer and Wan VAE pull automatically from the official base, and matrix-game verify rechecks an installed scene. Step 4, offline rendering proves the model produces output: matrix-game render --scene ancient-egypt --input lowpoly.mp4 --resources resources --preset realtime --prompt-mode fixed --output outputs/egypt, where the input is a low-poly proxy stream rather than real footage and the output is rendered.mp4 plus a configuration report with FPS and frame count preserved; the three presets matter - realtime is for A100, RTX 5090 switches to rtx5090, and quality is slower yet is exactly the default when --preset is omitted. Step 5, play and record locally without loading a model: bash scripts/build_game.sh then bash scripts/play.sh --scene ancient-egypt, giving the two shooter scenes a 48 FPS, 2x simulation-time demo mode with F9 recording and Escape menu, and MATRIX_GAME_DATA_DIR overriding the default data directory; racing uses build_racing.sh with 1x simulation time, and the browser demo does not enable the shooters' 2x mode either, advancing 1/--fps seconds per captured frame. Step 6, the browser demo splits server and client: mediamtx configs/mediamtx.yml in one terminal and CUDA_VISIBLE_DEVICES=0 bash scripts/serve.sh --scene ancient-egypt --resources resources --preset realtime --fps 40 in another, then open http://127.0.0.1:8025/, with --reference-only testing controls, HUD, audio and transport without loading a model; --fps 40 is a capture cadence target, not a throughput or latency guarantee. Step 7, the two RTX 5090 presets: rtx5090 needs no extra installation while rtx5090-sage requires building SageAttention 2.2.0 with the supplied CUDA-graph patch and CUDA 13.2 at git checkout eb615cf6cf4d221338033340ee2de1c37fbdba4a; both reuse the same scene checkpoints and shared resources, converting DiT block linear weights to FP8 at load with activations quantized during inference while embeddings and the output head stay BF16, and startup warms kernels and captures CUDA graphs so the first launch is slower than steady state. Step 8, acceptance in two forms: a keypress producing a picture in the browser demo (officially about under 150 ms, not independently reproduced) or an offline render turning a low-poly video into a finer output, with the three performance accounts written side by side (official 480p 60 FPS, 83 FPS and 18.5 GiB; the project page panel still at 0; and the README's about 38-39 FPS on an A100 at 864 x 480), remembering that model throughput, pipeline throughput, presentation frame rate and end-to-end latency are four different things. Step 9, six pitfalls: one process supports one session and one scene model; it binds to loopback by default and must not be exposed to the internet, since the official statement calls it a trusted single-user demo rather than a public cloud-gaming service; --fps 40 is not a throughput guarantee; Windows is unvalidated; disk must accommodate the Hub cache; and VRAM splits into an A100 80 GB reference tier and an RTX 5090 consumer tier with the 5080 and 5070 still pending.

Published October 16, 202610 min read
<!-- matrix-game-4-0-local-deploy-sop | sop | Can One GPU Run Matrix-Game 4.0? A Local Deploy SOP -->

Can a single GPU and one Linux box turn a "realtime playable" neural renderer from a demo clip into something running in your terminal? That promise has been made and broken repeatedly, so when Matrix-Game 4.0 shipped its code, weights and assets on 2026-10-15, the interesting question was not the advertised frame rate but a simpler one: if I follow the docs, will it run? This SOP answers exactly that: install from scratch on a Linux machine with an NVIDIA GPU, pull the resources, render one clip, open one browser demo, and know where you will trip.

Per the division of labor for batch 53, the companion open-source piece covers positioning, architecture, license and ecosystem. This piece covers install, run and verify only, and does not repeat the project background; nor does it cover realtime voice API integration.

Every command and flag below is taken verbatim from the v4 subdirectory README, from the SkyworkAI/Matrix-Game repository at Matrix-Game-4/README.md. Repository address: github.com/SkyworkAI/Matrix-Game

The nine steps below are the shortest path from a bare machine to "I press a key and the picture responds."

1. Align on the machine requirements first

The most common failure is not a typo but an environment assumption that does not hold.

Model inference, the real gate. You need Linux, an NVIDIA GPU and Python 3.12 at once. The reference A100 80 GB deployment uses PyTorch 2.12.0 / CUDA 13.2. The repo also ships a dedicated RTX 5090 runtime that does not auto-detect hardware: you must select the matching preset explicitly, or you fall through to the default path.

Browser demo. The server side must be a Linux host that also builds the game and the GStreamer pipeline. The desktop client targets Safari and Chrome and needs no GPU, so an ordinary laptop can open the page while the compute stays on the server.

Standalone play and recording. Needs macOS or Linux plus CMake, C++20, SDL2 and FFmpeg. The legacy Windows entry is retained but not validated for feature parity, so do not treat Windows as a production path.

Disk. Assets and weights are downloaded separately, not shipped in the repo, and the README warns to allow space for the Hugging Face Hub cache on top of the installed copy, so the same weights occupy disk twice.

2. Install inside the Matrix-Game-4/ directory

From Install onward, every command runs inside the Matrix-Game-4/ subdirectory, not the repository root. The root is a series repo and v4 is one subdirectory; enter the wrong one and everything after is wrong.

bash
git clone https://github.com/SkyworkAI/Matrix-Game.git
cd Matrix-Game/Matrix-Game-4
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.12.0 torchvision==0.27.0 --index-url https://download.pytorch.org/whl/cu132
python -m pip install -e '.[inference,server,download,test]'

Three things matter. The venv keeps the environment inside the project, and Python 3.12 is a hard requirement. The torch/torchvision pair comes from the official CUDA 13.2 wheel index and must stay paired. And -e '.[inference,server,download,test]' installs four extras; skip one and the matrix-game subcommands may fail with a missing module. The README adds that the optimized path needs version-matched PyTorch/Triton and a compatible driver, that FlashAttention is optional, and that only rtx5090-sage needs SageAttention. Dependency details live in docs/installation.md.

3. Download the resources: weights and assets come separately

Weights, the fast codec, sink images and game assets live on Hugging Face; the repo holds only manifests. The expected layout is:

text
resources/
  base/                        # Shared umT5, tokenizer, Wan VAE, TAEHV
  models/castle/                # model.safetensors, sink.png
  models/ancient-egypt/          # model.safetensors, sink.png
  models/racing/                # model.safetensors, sink.png
  game/assets/                  # Shooter assets and per-scene maps
  racing/assets/                # City map, captions, routes and vehicle audio

Game assets and model weights are two different manifests. Here is the README command, with the pinned repo id and revision:

bash
matrix-game download --manifest releases/game-assets.json \
  --repo-id Skywork/Matrix-Game-4.0 --repo-type model \
  --revision 4419accdb7bb7881234c5914de8cd915266e0325 \
  --scene ancient-egypt --resources resources

matrix-game download --manifest releases/models.json \
  --repo-id Skywork/Matrix-Game-4.0 --repo-type model \
  --revision 4419accdb7bb7881234c5914de8cd915266e0325 \
  --scene ancient-egypt --resources resources

Replace ancient-egypt with castle, racing or all to switch scenes; one server process supports only one scene model, so on-demand downloads save disk. The downloader verifies sizes and SHA256 against the pinned manifests, so an interrupted run can simply be restarted. The shared T5/tokenizer and Wan VAE are pulled automatically from a pinned official base-model revision, not duplicated here. To recheck an installed scene, use matrix-game verify with the same --manifest, --scene and --resources flags.

4. Offline rendering: confirm the model produces output

Once installation and downloads are done, do not rush to the browser. An offline render is the minimal validation because it decouples "can the model infer" from "network transport and browser decoding." The README suggests the fast path first: TAEHV encoding/decoding, one denoising step, a fixed scene prompt and the accelerated DiT runtime, selected with --preset realtime on A100.

bash
matrix-game render --scene ancient-egypt --input lowpoly.mp4 \
  --resources resources --preset realtime --prompt-mode fixed \
  --output outputs/egypt

The input is a low-poly proxy stream, not real footage: the game engine owns world state, physics and input and emits a clean low-poly RGB stream. The output is rendered.mp4 plus a configuration report, and FPS and frame count are preserved; only the final incomplete four-frame block is padded internally, then trimmed. The three presets differ: realtime (TAEHV, one-step denoising, ring/fused DiT, CUDA graph replay) is for A100, RTX 5090 switches to rtx5090, and quality (Wan VAE, two denoising steps 1000,500, native DiT) is slower and is the default when --preset is omitted. That last point is the trap: omitting the preset silently selects the slowest configuration.

5. Play and record locally without loading a model

This step loads no model weights and needs no Python-side model dependencies, so use it to confirm the game, controls and recording pipeline before you touch the model.

bash
bash scripts/build_game.sh
bash scripts/play.sh --scene ancient-egypt

For the two shooter scenes (Castle and Ancient Egypt), play.sh starts a 48 FPS / 2x simulation-time local demo mode, with no neural rendering and no simulated network delay. F9 toggles recording and Escape opens the menu. Use --scene castle for the other map, and set MATRIX_GAME_DATA_DIR to override the default resources/game. Racing is separate: bash scripts/build_racing.sh, then follow docs/racing.md; racing's 48 FPS recording mode uses normal 1x simulation time, unlike the shooters' 2x, and the browser demo also does not enable the shooters' local 2x mode, advancing simulation by 1 / --fps seconds per captured frame. A coding agent can also edit terrain, object placement, NPC behavior and game rules and re-test locally without loading model weights.

6. Browser demo: server and client are separate

The browser demo is a single-user WebRTC setup. Build the native game and install the media dependencies first, then open two terminals: one for MediaMTX, one for the model service.

bash
mediamtx configs/mediamtx.yml
bash
CUDA_VISIBLE_DEVICES=0 bash scripts/serve.sh \
  --scene ancient-egypt --resources resources --preset realtime --fps 40

Then open http://127.0.0.1:8025/ on the server, or forward the required ports over SSH as described in docs/online.md. On RTX 5090, replace --preset realtime with --preset rtx5090. To test controls, HUD, audio and transport without loading a model, add --reference-only, a handy switch for telling "is the model slow or is the network slow" apart. The README stresses that --fps 40 is a target capture cadence, not a throughput or latency guarantee: encoding, transport, buffering and browser decoding all add latency beyond model inference.

7. RTX 5090: two presets and optional SageAttention

After the standard installation and downloads, 5090 users start from --preset rtx5090. It reuses the same scene checkpoints and shared resources and needs no separate quantized checkpoint: the DiT block linear weights are converted to FP8 at load time with activations quantized during inference, while embeddings and the output head remain BF16.

Both 5090 presets select TAEHV, one-step denoising, fused kernels and CUDA graphs; they differ only in the self-attention backend:

PresetSelf-attentionAdditional setup
rtx5090PyTorch SDPANone beyond the standard installation
rtx5090-sageSageAttention for the full history window, SDPA while it fillsPatched build, below

To use rtx5090-sage, build SageAttention 2.2.0 with the supplied CUDA-graph patch and CUDA 13.2, from Matrix-Game-4/ in the same Python environment as the model:

bash
python -m pip install "nvidia-cuda-nvcc==13.2.*" "nvidia-nvvm==13.2.*" "nvidia-cuda-crt==13.2.*" "nvidia-cuda-cccl==13.2.*" setuptools wheel ninja
export CUDA_HOME="$(python -c 'import nvidia, os; print(os.path.join(nvidia.__path__[0], "cu13"))')"
mkdir -p ../cuda-link && ln -sf "$CUDA_HOME/lib/libcudart.so.13" ../cuda-link/libcudart.so
git clone https://github.com/thu-ml/SageAttention.git ../SageAttention
git -C ../SageAttention checkout eb615cf6cf4d221338033340ee2de1c37fbdba4a
git -C ../SageAttention apply "$PWD/matrix_game/inference/rtx5090_sage/sageattention.patch"
LIBRARY_PATH="$PWD/../cuda-link" MAX_JOBS=4 EXT_PARALLEL=2 python -m pip install --no-build-isolation ../SageAttention

Then replace --preset rtx5090 with --preset rtx5090-sage in either command above. No SageAttention build is needed for plain rtx5090. One note: the runtime warms up kernels and captures CUDA graphs before accepting a real session, so the first launch takes longer than steady-state rendering. Do not assume it has hung. A version for RTX 5080 and RTX 5070 is expected next week.

8. Acceptance criteria: what counts as "it works"

Two checkpoints. The browser demo path: you press a key and the picture responds, with an official end-to-end latency of about <150 ms, not independently reproduced. The offline render path: a low-poly video in, a finer rendered.mp4 out, with no network or browser involved and the cleanest acceptance test.

Three sets of figures must be written side by side; do not cherry-pick the best one.

  • Official release figures: one RTX 5090, 480p at 60 FPS, <150 ms end-to-end latency, 83 FPS steady single-step denoising and 18.5 GiB peak VRAM. Not independently reproduced.
  • The project page panel is currently a placeholder: 0 FPS / 0 ms / 0p / 0 GPU ("content will be updated at release").
  • The repository README's self-measured figures (the hardest source): one A100-SXM4-80GB at 864 x 480, one denoising step plus TAEHV plus SDPA plus ring/fused DiT plus CUDA graphs, excluding startup and warmup: about 38-39 FPS for the fixed-prompt shooters, about 38.5 for racing fixed and 28.9 for racing dynamic.

The README notes those numbers are neither browser presentation FPS nor action-to-photon latency, and that the racing figures come from an earlier checkpoint with the same architecture. As with --fps 40, these are different quantities: model-processing throughput, pipeline throughput, presentation frame rate and end-to-end latency.

9. Six pitfalls to know in advance

  1. One process supports only one active session and one scene model. Do not expect castle and racing in one server process, or multi-user concurrency.
  2. It binds to loopback by default; do not expose it to the internet ("trusted single-user demo, not an authenticated public cloud-gaming service").
  3. --fps 40 is capture cadence, not a throughput or latency guarantee; encoding, transport, buffering and browser decoding all add latency.
  4. The Windows entry is unvalidated and officially retained without feature-parity validation.
  5. Budget disk space, including the Hub cache; weights and assets download separately and the installed copy and cache add up.
  6. Two VRAM tiers: A100 80 GB for the reference route, RTX 5090 for the consumer route (5080/5070 pending).

FAQ

Q1: I do not have an A100. Can a consumer GPU run this? A: Yes, via the RTX 5090 route: after the standard installation, use --preset rtx5090 (PyTorch SDPA, no extra build). The README also mentions a rtx5090-sage variant requiring a SageAttention 2.2.0 build, and RTX 5080 and 5070 support is expected next week. For earlier cards, the README gives no support statement.

Q2: What is the difference between --preset realtime and --preset quality? A: realtime uses TAEHV, one-step denoising, ring/fused DiT and CUDA graphs and is the fast path on A100; quality uses Wan VAE, two denoising steps (1000,500) and native DiT and is slower. The key point: when --preset is omitted the default is quality, so "I did not write a preset" means the slowest option, not the fastest.

Q3: Why does play.sh work while the browser demo will not open? A: The paths differ. play.sh loads no model and needs no Python-side dependencies. The browser demo additionally needs MediaMTX running, GStreamer and the media dependencies, and the server must be the same Linux host. Start with --reference-only to test controls, HUD, audio and transport without loading a model.

Q4: The project page says 60 FPS while the README says 38 FPS. Which do I trust? A: Write both and label both. The 60 FPS / <150 ms / 83 FPS / 18.5 GiB set is the official release figure, not independently reproduced; the 38-39 FPS set is the README's self-measurement on an A100-SXM4-80GB at 864 x 480, excluding startup and warmup. Do not treat either as browser presentation FPS.

Q5: Can I port-forward this to the internet for friends to play? A: Not advisable. The README calls this a "trusted single-user demo," HTTP, RTSP and WebRTC bind to loopback by default, and there is no authentication. For remote use, forward the required ports over SSH (see docs/online.md).

Closing Thoughts

To recap: requirements (Linux + NVIDIA GPU + Python 3.12), installation inside Matrix-Game-4/, downloading weights and assets, an offline render, local play and recording, the browser demo, the 5090 presets, acceptance, and pitfall avoidance. The usual failure is never the command itself but three "default traps": commands must run in the subdirectory, omitting --preset selects the slowest tier, and --fps 40 is not a latency promise. Remember those three and the rest is mostly waiting for downloads and warmup. For everything else, read the companion open-source piece in this batch; the value of an SOP is saving the next person one evening.

This article is AI-assisted and human-edited. Last updated: 2026-10-16

FAQ

I do not have an A100. Can a consumer GPU run this?
Yes, via the RTX 5090 route: after the standard installation, use `--preset rtx5090` (PyTorch SDPA, no extra build). The README also mentions a `rtx5090-sage` variant requiring a SageAttention 2.2.0 build, and RTX 5080 and 5070 support is expected next week. For earlier cards, the README gives no support statement.
What is the difference between `--preset realtime` and `--preset quality`?
`realtime` uses TAEHV, one-step denoising, ring/fused DiT and CUDA graphs and is the fast path on A100; `quality` uses Wan VAE, two denoising steps (`1000,500`) and native DiT and is slower. The key point: **when `--preset` is omitted the default is `quality`**, so "I did not write a preset" means the slowest option, not the fastest.
Why does `play.sh` work while the browser demo will not open?
The paths differ. `play.sh` loads no model and needs no Python-side dependencies. The browser demo additionally needs MediaMTX running, GStreamer and the media dependencies, and the server must be the same Linux host. Start with `--reference-only` to test controls, HUD, audio and transport without loading a model.
The project page says 60 FPS while the README says 38 FPS. Which do I trust?
Write both and label both. The 60 FPS / <150 ms / 83 FPS / 18.5 GiB set is the **official release figure, not independently reproduced**; the 38-39 FPS set is the **README's self-measurement on an A100-SXM4-80GB at 864 x 480**, **excluding startup and warmup**. **Do not treat either as browser presentation FPS.**
Can I port-forward this to the internet for friends to play?
Not advisable. The README calls this a "trusted single-user demo," HTTP, RTSP and WebRTC bind to loopback by default, and there is no authentication. For remote use, **forward the required ports over SSH** (see `docs/online.md`).

Related

Field SOP

Run MiniCPM-V 4.7 on Your Phone: Three Paths

A hands-on SOP for running MiniCPM-V 4.7 from phone to desktop in three routes (all figures as of October 15, 2026, official pages prevail). First understand two official numbers: the 3 GB is the model's runtime memory footprint (the MoE-SSD mechanism keeps 35B expert weights on flash and streams them in on demand), not a device-RAM floor - the official app hardware table recommends 12 GB+ RAM for 35B-A3B (validated on the iPhone 17 Pro Max) and 4 GB+ for 1B, with smaller devices falling back to a smaller expert cache that runs noticeably slower; 13+ tokens/s is the official documentation figure for the iPhone 17 Pro Max, and other circulating figures (iPad 15+ and the like) are not adopted. Route 1, the official app on phone/tablet (OpenBMB/MiniCPM-V-Apps covers iOS, Android and HarmonyOS: TestFlight public link on iOS, APK from Releases on Android, signed HAP via DevEco on HarmonyOS; the 35B-A3B phone build is about 16.6 GB - main file plus 40 expert shards plus the vision module; 1B about 1.8 GB; the only fully offline route; pitfalls: slow storage throttles decoding, shards must match or the consistency check refuses to load, and the phone NPU stays unused short-term - the app runs the stock llama.cpp path). Route 2, desktop self-build with llama.cpp-omni for LES acceleration (the official LES link points to OpenBMB/llama.cpp-omni, a fresh repo created October 13 with 16 stars; the community tc-mb same-named repo has 291 stars, full docs and prebuilt packages - both share the same code lineage; verified commands: cmake -B build and cmake --build --target llama-omni-server/llama-omni-cli; GGUF weights in the official quantization repos on HF/ModelScope; honest limits: the README quick start targets the MiniCPM-o 4.5 full-duplex voice pipeline, 4.7 loading details are untested by this site and follow the official README). Route 3, the Ollama shortcut (the official library still lists only 4.6 as of October 15; the OpenBMB publisher page openbmb/minicpm-v4.7 runs today via ollama run - 1b about 1.8 GB, 35b-a3b Q4_K_M about 22 GB and the default latest tag, so small-memory machines must specify :1b explicitly; this route runs whole-model quants with no MoE-SSD, so "3 GB runs 35B" does not apply). One acceptance test for all three: ask about a local image and get a relevant answer. Division of labor with the same-batch open-source piece: that one covers capability and ecosystem, this one only covers getting it running. Repos: github.com/OpenBMB/MiniCPM-V and github.com/OpenBMB/MiniCPM-V-Apps.

Oct 15, 202610 min read
Field SOP

Running 27B on 4GB VRAM: Disk, Not VRAM, Stops You First

A five-step SOP for layer-wise inference with airllm (October 12, 2026; figures are vendor self-reported): (1) respect the triple version lock - torch must be a CUDA 12 build (no prebuilt flash-attn wheel for CUDA 13), plus flash-attn, plus a transformers branch that depends on the model (4.56.x for Kimi K3, 5.8+ for Qwen3.8-27B, an in-tree qwen4_exp build for Flash-Next), verified with both nvidia-smi and torch.cuda.is_available(); (2) pick a tier from the README measurements - the most realistic entry point is Qwen3.8-27B at 3.33GB (RTX 3090); (3) the first run pays for the download (Flash-Next checkpoint around 360GB, K3 in the TB class), finish it once and keep the cache directory, and reclaim pre-split originals with delete_original=True; (4) watch VRAM peaks with nvidia-smi and manage expectations - layer-wise streaming moves far more bytes than it computes, the vendor publishes no tokens-per-second figure so none is invented here, and slow is the design, not a bug; (5) wire it into agent or API workflows: fine for one-shot task-shaped work, wrong for token-by-token interactivity; training takes a different path (frozen weights streamed layer by layer, adapters kept on GPU). Five dedicated trap sections: the version matrix, the hundred-GB-to-TB first download that stalls, the hidden disk and host-RAM gates (64GB-class RAM to start), structurally slow output, and K3's hard flash-attn plus CUDA 12 constraints. Repo: github.com/lyogavin/airllm (Apache-2.0).

Oct 12, 202610 min read
Field SOP

Mellum2.1 Deploy SOP: Run a 12B Coding Agent on 24GB

A Mellum2.1 local deployment SOP (as of October 10, 2026; pairs with the open-source and comparison pieces). The vLLM main route in five steps: serve JetBrains/Mellum2.1-12B-A2.5B-Thinking, set reasoning-parser qwen3 (official model card; without it thinking text bleeds into answers), basic inference verification, tool calling (optional Hermes template) plus 131K-context validation, then an agent smoke test - each step with its expected result. Three traps first: the MTP head was not released with the weights, so the advertised 1.6x speedup is not reproducible today; GGUF is not out, so "announced" Ollama/LM Studio support is not usable today - distrust integration packs; and this is a thinking model, so a reasoning parser is mandatory. Hardware ledger: BF16 around the 24GB class (estimate; no official number), 16GB cards cannot hold it, 24GB (4090/3090) is the working floor and full-131K context adds KV cache on top. Two high-frequency failure modes get their own paragraphs: OOM dies at weight loading, not mid-generation (--gpu-memory-utilization only throttles KV cache, it cannot fit oversized weights), and first-run downloads of the large checkpoint can look hung - finish it once, keep the HF cache directory, and every later start is local. Dual-run reconciliation pairs with the comparison piece: race Qwen3.5-9B on the same harness for tokens/s and tickets solved; performance conclusions come from your own repo, not vendor charts. Repo: huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking.

Oct 10, 20269 min read