中/EN
Field SOP
Field SOP

Tencent 770B Flagship Self-Hosting SOP: Eight H100s Can't Even Fit the FP8 Weights - Official Baseline Is 16x B200

A complete SOP for self-hosting Tencent's 770B flagship Hy4 preview. Cold water first, via the VRAM math: FP8 weights run about 770GB, and the official vLLM recipe states the baseline is 16xB200 or 8xB300 (weights + KV cache) - 8xH100 (640GB) cannot even fit FP8 weights, since 49B active params save compute but all 770B weights must sit in VRAM. Both deployment routes quote the official README verbatim: the vLLM prebuilt image (MTP speculative decoding num_speculative_tokens=3, FLASHMLA_SPARSE attention backend, hy_v4 tool/reasoning parsers) and the SGLang prebuilt image (NEXTN speculation, tp-size 8). Includes OpenAI-compatible invocation (temperature 0.9 / top_p 1.0; no_think to skip deep reasoning and save output tokens), AngelSlim self-quantization, the finetune pipeline, 7 pitfalls and a 10-item launch checklist; if you skip self-hosting, use Tencent Cloud TokenHub/OpenRouter or the two-week free tier on WorkBuddy/CodeBuddy.

Published August 29, 202612 min read
<!-- hy4-preview-self-hosting-sop | sop | Tencent 770B Flagship Self-Hosting SOP: Eight H100s Can't Even Fit the FP8 Weights - Official Baseline Is 16x B200 -->

On 2026-08-28, Tencent Hunyuan launched and open-sourced its new-generation flagship Hy4 preview: a 770B-total / 49B-active MoE with 1M context, Apache 2.0 license, and dual BF16/FP8 weights across four distribution channels (launch background and architecture teardown in our Hy4 preview coverage).

This SOP answers one concrete question: what does it take to run Hy4 preview yourself, how do you get it running, and which pitfalls will burn an afternoon? All deployment commands and parameters below come from the official GitHub README and the official vLLM/SGLang recipes (verified 2026-08-29); memory figures are engineering estimates, not official commitments.

Three boundaries up front. First, this is not an "anyone can run it" tutorial - 770B is the top tier of currently open weights; check Step 0 for the hardware bar and go straight to the API if you don't clear it. Second, the official README's command template says --tensor-parallel-size 8, but that assumes a single node of B300-class memory, not eight 80GB cards - this misunderstanding is so common it gets pitfall #1. Third, this is not an official partnership or promotion; the license and weight links are governed by the official repo.

Step 0: Do the Memory Math Before You Talk Deployment

The first gate for self-hosting isn't technical, it's arithmetic. The MoE iron rule: the 49B activation saves per-token compute, not memory - all 770B of weights must reside in GPU memory. Back-of-envelope by bytes per parameter (engineering estimate, excluding KV cache and runtime overhead):

PrecisionBytes per paramWeight memory (est.)Official recipe hardware baseline
BF162~1.5 TBReference tier for long-context / full-precision runs in the official recipe
FP81~770 GB16x B200 or 8x B300 (with MTP) minimum

The official vLLM recipe page for Hy4 preview states verbatim: "16x B200 or 8x B300 minimum for weights + KV cache". Note this is the floor for weights plus KV cache, not a comfortable margin - KV cache for 1M context under concurrency is similarly substantial, and the recipe further notes that filling the full 1M context requires an 8x B300 node or 4 GB300 NVL4 trays (3 trays physically hold it, but TP12 is invalid - 64 attention heads are not divisible by 12).

In procurement language: 8x H100 (640GB) cannot even fit the FP8 weights (~770GB). If your cluster is 80GB-class cards, the conclusion isn't "tune some flags" - it's "this road is closed." Go straight to the API on Tencent Cloud TokenHub or OpenRouter (OpenRouter snapshot pricing roughly $0.834/M input, $2.501/M output, as of 2026-08-28), or wait for community-quantized builds. For compliance scenarios where data can't leave the premises, consider renting B200/B300-class compute rather than squeezing an existing cluster.

Self-hosting suits two kinds of users: hard data-sovereignty constraints, and batch-inference volumes whose pay-as-you-go bills already exceed cluster amortization. If neither applies, you can skip the second half of this article.

How to compute the amortization threshold (estimate framework): monthly self-hosting cost = (hardware rental or depreciation + power and ops) ÷ monthly tokens processed, giving a "composite cost per million tokens" to compare against API list prices. Using OpenRouter snapshot pricing ($0.834/M input, $2.501/M output, as of 2026-08-28) as the reference: for monthly volume I (input) and O (output) in millions of tokens, the API bill is roughly 0.834×I + 2.501×O; plug your B200/B300 node quote into the self-hosting side, and the intersection of the two curves is your break-even volume. Empirically, only sustained full-node batch workloads (seven-day weeks at high utilization) reach that point; stop-and-go weekday loads never beat the API. If you don't have a quote yet, build the formula in a spreadsheet and wait for one - that half-hour beats buying GPUs on a hunch.

Three Things to Prepare Before Deployment

The command line is the last step; without the following three, docker run just exits in a loop:

  1. Weights and disk: the FP8 build runs to hundreds of GB; reserve at least 1.5x for download cache and extraction (the Hugging Face cache mechanism creates temporary copies). In mainland-China environments prefer ModelScope or CNB; on Hugging Face, configure an HF_ENDPOINT mirror or use hf_transfer for acceleration. Prove your download script on 10GB before unleashing the full run - at 770B scale, one interrupted download costs hours.
  2. Container and drivers: both official routes use docker run --gpus all; the prerequisites are an NVIDIA driver plus nvidia-container-toolkit. Verify GPU visibility in containers first with docker run --rm --gpus all nvidia/cuda nvidia-smi. Multi-node deployments also need inter-node networking and shared storage sorted - don't download the weights separately on every machine.
  3. Service acceptance: a running container is not a ready model - loading 770B of weights takes minutes on its own. Use curl http://127.0.0.1:8000/v1/models to confirm the service is registered under the name hy4-preview before sending the first chat. Don't wire up upstream services before the health check passes; this is the common root cause of every "container started fine, API calls 502" incident.

Route 1: The Official vLLM Image

The vLLM command template from the official README (quoted verbatim, verified 2026-08-29):

bash
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-preview

Five flags, each worth understanding before you touch anything:

  1. vllm/vllm-openai:hy4-preview is the official prebuilt image: don't fight the generic latest image with hand-installed dependencies. Gated DSA, IndexCache, MTP, and the hy_v4 parsers are all new; a version mismatch buys you blank screens and tracebacks. The official recipe requires vLLM 0.29.0+.
  2. --speculative-config enables MTP speculative decoding: Hy4 ships with 1 native MTP layer (10B total / 0.7B active); num_speculative_tokens: 3 drafts 3 tokens per step. This is a key link in making a 770B model deliver usable throughput - don't delete it for tidiness.
  3. --attention-backend FLASHMLA_SPARSE: this matches the Gated DSA (a gated variant of DeepSeek Sparse Attention) plus IndexCache cross-layer sparse-index-reuse attention architecture; only the dedicated backend gets you real performance.
  4. hy_v4 for both parsers: tool-call and reasoning both use hy_v4, and --enable-auto-tool-choice is required for tool calls and thinking content to come out correctly. Swapping models without swapping parsers is the most common source of "it runs, but the output is garbage."
  5. What --tensor-parallel-size 8 really means: TP must divide the 64 attention heads (8 and 16 work; 12 does not). The official template assumes FP8 weights on 8 cards with B300-class per-card memory (~288GB); 16x B200 also works. 80GB-card clusters should jump straight to pitfall #1.

Route 2: The Official SGLang Image

Teams standardized on SGLang take this route; the official image is multi-arch (x86 and Arm):

bash
docker pull lmsysorg/sglang:hy4-preview

docker run --gpus all --ipc=host -p 8000:8000 lmsysorg/sglang:hy4-preview \
  python3 -m sglang.launch_server \
    --model tencent/Hy4-preview-FP8 \
    --tp-size 8 \
    --reasoning-parser auto \
    --tool-call-parser auto \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --port 8000 \
    --served-model-name hy4-preview

Item-for-item this mirrors the vLLM version: NEXTN is MTP speculative decoding (next-n), and speculative-num-steps 3 aligns with vLLM's num_speculative_tokens: 3. The parser flags differ between the two (SGLang uses auto, vLLM uses hy_v4) - copy each template as written and never cross-mix. Both routes expose a standard OpenAI-compatible service with identical client code - pick by your team's existing serving stack and stop agonizing.

First Call

With the service up on 127.0.0.1:8000, the minimal example from the official README (quoted verbatim):

python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
)
print(response.choices[0].message.content)

Two inference details you must know:

  • Recommended parameters are temperature=0.9, top_p=1.0. Unlike the 0.7/0.9 habits carried over from other model families, the official docs recommend this pair explicitly; establish your baseline with them before customizing.
  • Reasoning mode defaults to "high" (deep chain-of-thought), suited to math, coding, and complex reasoning. For direct answers (fewer thinking tokens, lower latency), pass extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}. Note self-hosting has no "half-price thinking tokens" - thinking time burns your own GPUs, and under high concurrency turning thinking off is throughput you can bank.

Discipline for the 1M window (engineering estimate): however large you open the context window, the KV cache claims that much memory budget. The official 16x B200 / 8x B300 line is "weights + KV cache" minimum, which means headroom for concurrency is already thin at full 1M context. The right posture in production is setting --max-model-len to your traffic's real distribution: most chat and coding workloads need a few thousand to tens of thousands of tokens; a 1M window opened "just in case" trades concurrency for psychological comfort. For genuinely long documents, run retrieval and segmentation first and let the model read the results - the same discipline from our LLM API Cost Optimization SOP, restated for the self-hosting side.

Level Up: Quantization and Finetuning

  • Quantization: the official AngelSlim compression toolkit covers common quantization algorithms, low-bit quantization, and speculative sampling. If you want to compress your own deployment tier, start from the official toolchain rather than hand-rolled calibration.
  • Finetuning: the repo ships a complete finetuning pipeline (finetune/README.md). Full-parameter finetuning of 770B is not a personal-budget project; the exact level of official pipeline support is documented in the repo - read it before starting.
  • Weight downloads: BF16 and FP8 are both on Hugging Face, ModelScope, GitCode, and CNB. In mainland-China environments prefer ModelScope or CNB; at 770B-scale download volumes, one tier of network path difference costs a day.

Seven Typical Pitfalls

  • Forcing it onto 8x H100 - the official recipe says it plainly: 16x B200 or 8x B300 minimum for weights + KV cache. 8x 80GB (640GB) can't even hold the ~770GB of FP8 weights, and no amount of --gpu-memory-utilization tuning saves you. The README's TP=8 assumes B300-class per-card memory.
  • Picking a TP that doesn't divide the head count - 64 attention heads; TP must be a sensible factor (8, 16). The recipe specifically flags TP12 as invalid: 3 GB300 trays can physically hold the weights, but TP12 simply won't start.
  • Deleting the speculative-decoding config to "save resources" - the MTP layer is built into the architecture; removing speculative-config/NEXTN costs a chunk of throughput while saving almost no memory. Strictly negative ROI.
  • Mismatched parsers - vLLM uses hy_v4, SGLang uses auto; copy each stack's template. Garbled tool-call output is eight times out of ten a parser mismatch.
  • Using the generic latest image - Gated DSA / FLASHMLA_SPARSE / MTP support requires the official dedicated images (vLLM 0.29.0+); generic images mostly fail to start or run hobbled.
  • Defaulting to BF16 - BF16 weights (~1.5TB) are double FP8; unless you have a specific full-precision need (quantization baselines, reproduction runs), deploy the official FP8 and spend the savings on KV cache and concurrency.
  • No download plan - 770B-class weights run to hundreds of GB per precision, doubling if you pull both. Confirm the channel (ModelScope/CNB in China), disk, and bandwidth before starting.

Pre-Launch Checklist

  1. Check memory against the table: FP8 ~770GB floor; your node is 16x B200 / 8x B300 class - 80GB clusters walk away
  2. Use the official prebuilt images (vllm/vllm-openai:hy4-preview or lmsysorg/sglang:hy4-preview), vLLM 0.29.0+
  3. MTP speculative decoding config retained (vLLM speculative-config / SGLang NEXTN)
  4. Attention backend FLASHMLA_SPARSE in effect
  5. Parsers configured per stack: vLLM hy_v4, SGLang auto
  6. --served-model-name hy4-preview matches the client's model field
  7. Recommended parameters applied: temperature=0.9, top_p=1.0
  8. Reasoning-mode policy is explicit: default high for complex tasks, no_think for high-concurrency direct-answer serving
  9. TP value divides 64 heads; multi-node plans checked against the recipe's TP-validity notes
  10. Weight channel, disk, and bandwidth planned in advance; FP8 first

One-line close: Hy4 preview raises the open-source flagship's top tier to 770B - and its first message to self-hosters isn't a command line, it's arithmetic: confirm your cluster can hold 770GB before debating vLLM vs SGLang; if it can't, the API is your self-hosting.

FAQ

Q1: Can 8x H100 really not run Hy4 preview? The README command literally says TP=8. A1: It cannot. The README template's --tensor-parallel-size 8 corresponds to the hardware baseline in the official recipe - for the FP8 model, "16x B200 or 8x B300 minimum for weights + KV cache." 8x H100 totals 640GB, below the ~770GB estimated floor of the FP8 weights alone, so the model OOMs during loading; smaller batches or lower gpu-memory-utilization won't save it. 80GB-card clusters should use the API or wait for community-quantized builds.

Q2: BF16 or FP8 - which version should I pick? A2: Default to the official FP8: ~770GB of weights, half of BF16 (~1.5TB), with the savings converting directly into KV cache and concurrency. BF16 fits two scenarios: quantization-baseline comparisons and reproductions with hard precision requirements. Both versions are open on Hugging Face, ModelScope, GitCode, and CNB.

Q3: Does no_think hurt intelligence, and what does it save? A3: The official positioning of reasoning_effort: "no_think" is direct-answer scenarios, skipping the deep chain-of-thought. Keep the default high for math, coding, and complex reasoning; for simple Q&A and high-concurrency direct answers, turning it off meaningfully cuts latency and frees throughput. In self-hosting you're saving your own GPU time - under high concurrency that throughput gain is more tangible than in API scenarios. Benchmark both settings on your real tasks before committing.

Q4: vLLM or SGLang? A4: Officially both are first-class routes, both expose OpenAI-compatible services with zero client-code difference. Choose by your team's existing ops stack: on vLLM, use the vllm/vllm-openai:hy4-preview image; on SGLang, use lmsysorg/sglang:hy4-preview (multi-arch, x86/Arm). The parser flags differ (hy_v4 vs auto) - copy each template as written, never cross-mix.

Q5: I want to finetune Hy4 preview - realistic for an individual? A5: The repo provides a complete finetuning pipeline, but full-parameter finetuning at 770B is a multi-node-cluster project outside personal budgets. The realistic individual path: validate capability via the API, and for lighter needs look at quantized deployment tiers built with the official AngelSlim toolchain, or await community distills/smaller variants. Specific finetuning support is documented in the repo's finetune/README.md.

References

Deployment commands and hardware baselines are a 2026-08-29 snapshot of official documentation, not an official partnership or promotion; memory figures are engineering estimates - defer to the official recipes and your own measurements.

This article is AI-assisted and human-edited. Last updated: 2026-08-29

FAQ

Can 8x H100 really not run Hy4 preview? The README command literally says TP=8.
It cannot. The README template's `--tensor-parallel-size 8` corresponds to the hardware baseline in the official recipe - for the FP8 model, "16x B200 or 8x B300 minimum for weights + KV cache." 8x H100 totals 640GB, below the ~770GB estimated floor of the FP8 weights alone, so the model OOMs during loading; smaller batches or lower `gpu-memory-utilization` won't save it. 80GB-card clusters should use the API or wait for community-quantized builds.
BF16 or FP8 - which version should I pick?
Default to the official FP8: ~770GB of weights, half of BF16 (~1.5TB), with the savings converting directly into KV cache and concurrency. BF16 fits two scenarios: quantization-baseline comparisons and reproductions with hard precision requirements. Both versions are open on Hugging Face, ModelScope, GitCode, and CNB.
Does `no_think` hurt intelligence, and what does it save?
The official positioning of `reasoning_effort: "no_think"` is direct-answer scenarios, skipping the deep chain-of-thought. Keep the default `high` for math, coding, and complex reasoning; for simple Q&A and high-concurrency direct answers, turning it off meaningfully cuts latency and frees throughput. In self-hosting you're saving your own GPU time - under high concurrency that throughput gain is more tangible than in API scenarios. Benchmark both settings on your real tasks before committing.
vLLM or SGLang?
Officially both are first-class routes, both expose OpenAI-compatible services with zero client-code difference. Choose by your team's existing ops stack: on vLLM, use the `vllm/vllm-openai:hy4-preview` image; on SGLang, use `lmsysorg/sglang:hy4-preview` (multi-arch, x86/Arm). The parser flags differ (`hy_v4` vs `auto`) - copy each template as written, never cross-mix.
I want to finetune Hy4 preview - realistic for an individual?
The repo provides a complete finetuning pipeline, but full-parameter finetuning at 770B is a multi-node-cluster project outside personal budgets. The realistic individual path: validate capability via the API, and for lighter needs look at quantized deployment tiers built with the official AngelSlim toolchain, or await community distills/smaller variants. Specific finetuning support is documented in the repo's `finetune/README.md`.

Related

Field SOP

Qwen3.8-Flash-Next Full-Stack Deployment SOP: 125B Main Model plus 51B N-gram Embeddings, Three Tiers from Hosted API to Apple Silicon

A three-tier route for taking Qwen3.8-Flash-Next from "it runs" to "it runs cheaply". The managed tier needs no ops: the QwenCloud API speaks both OpenAI and Anthropic specs, and QwenWork's Standard mode is powered by this model. For self-hosted serving, four commands quoted verbatim from the official README: transformers serve (--continuous-batching), SGLang (--tp-size 4 --context-length 262144 --reasoning-parser qwen3 --tool-call-parser qwen3_coder), vLLM (--tensor-parallel-size 4 --max-model-len 262144 --enable-auto-tool-choice) and TokenSpeed, all exposing an OpenAI-compatible API at localhost:8000/v1. Local and edge paths include GGUF builds via llama.cpp, mlx-vlm on Apple Silicon, and Unsloth. The engineering detail most worth remembering: the extra 51B of N-gram embeddings can be offloaded to host memory and overlapped with model compute through async prefetch. Because the README gives no official VRAM baseline, this SOP refuses to guess a hardware floor and marks it as "defer to the official recipe and your own measurements". Also covers the trade-offs of YaRN extrapolation to 1M, fine-tuning framework choices (Unsloth, Swift, Llama-Factory) and seven pitfalls - the first being that the GitHub repo ships no LICENSE file, so check the model page before commercial use.

Aug 30, 202612 min read
Field SOP

Self-Hosting OpenMAIC: From Zero-Deploy to Agent Workbench

A complete SOP for getting OpenMAIC running from zero: (1) zero-deploy hosted mode with an access code from open.maic.chat; (2) standard local setup (pnpm >= 10: clone, pnpm install, .env, pnpm dev); (3) production (pnpm build && pnpm start, one-click Vercel, docker compose up --build); (4) advanced (Postgres persistence profile, ACCESS_CODE, MP4 export profile, Lemonade/FunASR local providers); (5) wiring it into agent workbenches (clawhub install openmaic or importing skills/openmaic/, generating classrooms from Feishu/Slack messages). Includes 6 pitfalls and a 10-item pre-launch checklist, with every command copied verbatim from the official README.

Sep 8, 202611 min read
Field SOP

Build Long-Running Agent Workflows with GPT-6 Astra

A hands-on SOP for building long-running agent workflows on GPT-6 Astra's real capabilities (1.05M context, 128K output, 0% alignment overreach): start with three prerequisites (OpenAI Python SDK 1.50+, the OPENAI_API_KEY environment variable, and API allowlist), then proceed in order through long-context planning, tool definition (function calling plus computer use), async invocation, mid-flight correction, and acceptance with cost control. Key points: on the first call place only the goal, acceptance criteria, tool list, and key background so the model emits a plan first; tools must specify name, description, and parameters; use streaming events plus a background queue and task-id polling for async; correct course by injecting new instructions without restart; and accept only via independent assertion scripts while keeping max_output_tokens small and setting a daily spend cap.

Sep 4, 202611 min read