Field SOP
Field SOP

Local LLM Deployment with Ollama: An SOP on Quantization, VRAM, and Pitfalls

A full SOP for taking Ollama local LLMs from "can run" to "usable": why quantization (q4_K_M/q8_0) decides model intelligence, why KV Cache makes you slower the longer you chat, the hardware passing line, picking models by task, Open-WebUI deployment + Ollama API automation (with Python code + JSON mode), 6 pitfalls + FAQ. Speed figures are community reference magnitudes, not hands-on.

Published August 5, 20269 min read
<!-- ollama-local-llm-deploy-sop | sop | Local LLM Deployment with Ollama: An SOP on Quantization, VRAM, and Pitfalls -->

Many people install Ollama to run local large models and get disillusioned in 10 seconds: you type ollama run, the first Q&A comes back instantly, the second slightly complex question starts spitting out one word at a time, the fan takes off, and the answer is a mess. The "one-click deploy, free unlimited" tutorials only get you on board; they do not tell you there is a gulf of quantization and VRAM between "can run" and "usable".

This SOP lays out the real pitfalls of local LLM deployment: why quantization decides model intelligence, why VRAM bandwidth makes you "slower the longer you chat", what hardware config is the passing line, and how to wire local models into Open-WebUI and automation scripts. The tool focus is Ollama (lightweight, one command), with vLLM (production throughput) for the advanced tier. Speed figures here are community multi-source reference magnitudes, not hands-on testing; actual performance depends on your hardware.


One: Quantization -- Model Size Is Not Intelligence

You think you are running a 14B-parameter model, but you are actually running its "compressed biscuit" version. Most models Ollama pulls by default carry a quantization tag: q4_K_M means weights are compressed from 16-bit floats to 4-bit, the file is 4x smaller, at the cost of precision. Quantization level is the key variable that decides local model intelligence, not parameter count.

Common quantization tiers:

QuantBitsSizePrecisionUse case
q4_K_M4-bitSmallestClearly degradedVRAM-tight, need larger params
q5_K_M5-bitSmallMildly degradedCompromise
q8_08-bitLargerNear originalEnough VRAM, want quality
f1616-bitLargestOriginalHome GPUs basically cannot run

Community reference: the same model at q8_0 scores over ten percentage points higher on coding-task pass rates than q4_K_M (multi-source, not hands-on, per actual). A 7B at q4_0 has 75% of its precision stripped; do not expect it to handle complex business logic. Check the quantization level first when picking a model.

bash
# Pull a model (tag per the Ollama model library)
ollama pull qwen2.5:14b

# List installed models
ollama list

# Run
ollama run qwen2.5:14b

Available quantization tags per model are on the ollama.com/library model page; different models support different tags.


Two: VRAM and KV Cache -- Why It Gets Slower the Longer You Chat

"My MacBook has 32GB of memory, that's plenty for a 14B, right?" In theory yes, but multi-turn chat gets progressively slower. The culprit is the context window: each turn, the model has to cram all prior history into VRAM/memory and recompute, and this KV Cache grows linearly with turns. Once context exceeds a few thousand tokens, memory bandwidth becomes the bottleneck and speed falls off a cliff.

Community reference magnitudes (multi-source, not hands-on, per actual hardware):

HardwareModelQuantShort chat (tok/s)Long context (tok/s)Production-usable
32GB unified memory (Mac)qwen2.5:14bq4_K_M~15~3Long-chat collapses
32GB unified memory (Mac)qwen2.5:7bq8_0~28~18Acceptable
12GB VRAM (RTX 4070 Ti)llama3.1:8bq6_K~65~48Smooth
Dual 4090 48GBqwen2.5:72bq4_K_M~35~28Real productivity

Conclusion: a 32GB Mac running a 14B collapses in long-chat scenarios. What you lack is not a GPU, it is VRAM bandwidth and capacity. This is also where vLLM adds value -- it manages KV Cache highly efficiently with PagedAttention, supporting longer context on the same hardware. But vLLM needs Linux plus CUDA, so the deployment bar is higher. Ollama is positioned for lightweight dev/testing, not production-grade inference.


Three: The Hardware Passing Line

Direct conclusion -- this is the "passing line" for local deployment:

  1. 7B-8B models (llama3.1, qwen2.5): minimum 16GB unified memory or VRAM, 24GB recommended, quant q6_K or q8_0. Below this you are running a toy.
  2. 14B-16B models (qwen2.5:14b, deepseek-coder): minimum 32GB unified memory or 12GB VRAM, 48GB or 24GB recommended, q4_K_M is the floor.
  3. 70B+ models: a single home GPU cannot run them; you need dual 4090s or a Mac Studio M2 Ultra (192GB).

Do not run a 7B on a Raspberry Pi and call it "usable" -- 0.5 tok/s is not usable.


Four: Model Selection Strategy -- Pick by Task, Do Not Just Pull the Latest

Not every task needs a large model. Selection advice:

  • Code generation: deepseek-coder-v2:16b (q6_K, needs 12GB VRAM) or qwen2.5-coder:7b (q8_0, 8GB works). The former is stronger on complex tasks; the latter is faster, good for completion.
  • General chat/writing: llama3.1:8b (q6_K, stronger English) or qwen2.5:14b (q4_K_M, smoother Chinese).
  • Translation/summarization: 7B is enough; qwen2.5:7b (q8_0) can even match a 14B quantized version on translation, because quantization hurts "precise output" tasks more.
  • RAG/knowledge Q&A: 14B minimum, quant at least q5_K_M; smaller models or low quantization will frequently misattribute.

Five: Practice -- Open-WebUI + Ollama API

The real value of a local model is wiring it into your toolchain, not just chatting in a terminal.

Build a local AI workstation with Open-WebUI

Ollama ships with a CLI, but Open-WebUI gives you a ChatGPT-like interface with multi-model switching, history management, and document RAG:

bash
# Pull the image
docker pull ghcr.io/open-webui/open-webui:main

# Start (mount the data dir so you do not lose data on restart)
mkdir -p ~/open-webui/data
docker run -d -p 3000:8080 \
  -v ~/open-webui/data:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 in a browser, set the Ollama address in settings (default http://localhost:11434), and you can use local models in a GUI.

Automation with the Ollama API

Once Ollama is running it exposes a REST API by default (port 11434), callable from any language:

python
import requests
import json

# Non-streaming call
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "qwen2.5:14b",
        "prompt": "Translate the following into English: 今天天气真好",
        "stream": False,
        "options": {"num_ctx": 2048, "temperature": 0.1},
    },
)
print(response.json()["response"])

# Structured output (JSON mode -- key to using local models in dev)
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "qwen2.5:14b",
        "prompt": "Extract the company name and amount as JSON:\nParty A Beijing Tech Co., contract amount 5 million yuan",
        "stream": False,
        "format": "json",  # forces valid JSON output
        "options": {"temperature": 0},
    },
)
print(json.loads(response.json()["response"]))

format: "json" is the key to using local models in development: it forces valid JSON output for information extraction, auto-tagging, batch processing. The correct use of local models is high-frequency, simple, batchable tasks.


Six: Pitfall Log

Pitfall 1: Treating a 7B q4_0 as ChatGPT. With 75% of precision stripped, complex tasks will fail. A local model is not a free ChatGPT replacement; it is a supplement.

Pitfall 2: Not clearing context as chat slows. KV Cache grows with turns; long chats will collapse. Type /clear in the Ollama session to wipe history, or /set parameter num_ctx 2048 to cap context length.

Pitfall 3: Not releasing VRAM when full. On Windows, when VRAM fills up, run ollama stop and restart, or restart the Ollama service to fully clear VRAM.

Pitfall 4: C-drive filling up. Ollama stores models on the C drive by default. Point OLLAMA_MODELS at another drive; do not wait until C is full.

Pitfall 5: Not knowing what quantization you are running. Run ollama list, compare file sizes on ollama.com/library -- for the same param count, a smaller file means lower intelligence.

Pitfall 6: Taking Ollama to production. Ollama is a lightweight dev/test tool with limited throughput. For production-grade concurrency pick vLLM (PagedAttention plus continuous batching), but it needs Linux plus CUDA.


FAQ

Q1: Can local deployment replace ChatGPT? A: For most people, no. For everyday chat, email, and lookup, cloud models (ChatGPT, Claude, DeepSeek) beat local -- larger models, faster inference, stronger multimodal. A local model is a supplement, not a replacement.

Q2: How big a model can a MacBook run? A: 32GB unified memory can run 7B (q8_0, smooth) up to 14B (q4_K_M, short chat OK, long chat collapses). 16GB only fits 7B. You need a 192GB Mac Studio M2 Ultra for 70B.

Q3: Ollama or vLLM? A: Ollama is simple, one command, good for dev/testing and single-user. vLLM has high throughput and manages VRAM efficiently with PagedAttention, good for production-grade concurrency, but needs Linux plus CUDA. Personal use: Ollama; at scale: vLLM.

Q4: What are local models actually good for? A: Four scenarios: extreme privacy (law, medical -- data that cannot go to the cloud), offline environments (planes, classified sites), high-frequency automation (batch processing, zero marginal cost), and experimental freedom (custom system prompts, tuning, merging LoRA).

Q5: Which quantization level? A: If VRAM is ample, q8_0 (near-original quality); if tight, q4_K_M (runs but clearly degraded); compromise at q5_K_M. For precision tasks like code and RAG, do not go below q5_K_M.


Take

The truth of local LLM deployment: it is not a free ChatGPT, but a tool you can only wield if you understand quantization, VRAM, and inference frameworks. Ollama lowers the bar to "can install software", but between "can run" and "usable" lies a technical gulf. Most people's disillusionment with local models is not Ollama's fault -- it is the expectation mismatch of treating a 7B q4_0 as GPT-4.

If you have a hard privacy requirement, offline needs, or high-frequency automation tasks, local models are worth the effort -- pick the right quantization, provision enough VRAM, use the right scenario, and it goes from "conks out in 10 seconds" to steady output. If you just use AI day to day, $20/mo ChatGPT Plus is honestly better. Do not be kidnapped by "free"; choose by need.


References

This article is AI-assisted and human-edited. Last updated: 2026-08-05

FAQ

Can local deployment replace ChatGPT?
For most people, no. For everyday chat, email, and lookup, cloud models beat local - larger models, faster inference, stronger multimodal. A local model is a supplement, not a replacement.
How big a model can a MacBook run?
32GB unified memory can run 7B (q8_0, smooth) up to 14B (q4_K_M, short chat OK, long chat collapses). 16GB only fits 7B. You need a 192GB Mac Studio M2 Ultra for 70B.
Ollama or vLLM?
Ollama is simple, one command, good for dev/testing and single-user. vLLM has high throughput and manages VRAM efficiently with PagedAttention, good for production-grade concurrency, but needs Linux plus CUDA. Personal use: Ollama; at scale: vLLM.
What are local models actually good for?
Four scenarios: extreme privacy (law, medical - data that cannot go to the cloud), offline environments (planes, classified sites), high-frequency automation (batch processing, zero marginal cost), and experimental freedom (custom system prompts, tuning, merging LoRA).
Which quantization level?
If VRAM is ample, q8_0 (near-original quality); if tight, q4_K_M (runs but clearly degraded); compromise at q5_K_M. For precision tasks like code and RAG, do not go below q5_K_M.

Related

Field SOP

Weights Downloaded, Now What? A VRAM Planning Guide

Between "weights downloaded" and "model running" sits a VRAM bill: weight memory equals parameters times bytes per parameter (BF16/FP8/INT4), and per-token KV cache equals 2 x layers x KV heads x head_dim x bytes. Worked examples span Qwen3-32B (66GB down to 17GB across precisions) and K3.1 Pro (5.6TB BF16), plus three misjudgments, precision-by-GPU guidance (24GB/96GB/Blackwell), vLLM and Ollama toolchains, and a five-step landing checklist.

Nov 4, 20269 min read
Field SOP

Run MiniCPM-V 4.7 on Your Phone: Three Paths

A hands-on SOP for running MiniCPM-V 4.7 from phone to desktop in three routes (all figures as of October 15, 2026, official pages prevail). First understand two official numbers: the 3 GB is the model's runtime memory footprint (the MoE-SSD mechanism keeps 35B expert weights on flash and streams them in on demand), not a device-RAM floor - the official app hardware table recommends 12 GB+ RAM for 35B-A3B (validated on the iPhone 17 Pro Max) and 4 GB+ for 1B, with smaller devices falling back to a smaller expert cache that runs noticeably slower; 13+ tokens/s is the official documentation figure for the iPhone 17 Pro Max, and other circulating figures (iPad 15+ and the like) are not adopted. Route 1, the official app on phone/tablet (OpenBMB/MiniCPM-V-Apps covers iOS, Android and HarmonyOS: TestFlight public link on iOS, APK from Releases on Android, signed HAP via DevEco on HarmonyOS; the 35B-A3B phone build is about 16.6 GB - main file plus 40 expert shards plus the vision module; 1B about 1.8 GB; the only fully offline route; pitfalls: slow storage throttles decoding, shards must match or the consistency check refuses to load, and the phone NPU stays unused short-term - the app runs the stock llama.cpp path). Route 2, desktop self-build with llama.cpp-omni for LES acceleration (the official LES link points to OpenBMB/llama.cpp-omni, a fresh repo created October 13 with 16 stars; the community tc-mb same-named repo has 291 stars, full docs and prebuilt packages - both share the same code lineage; verified commands: cmake -B build and cmake --build --target llama-omni-server/llama-omni-cli; GGUF weights in the official quantization repos on HF/ModelScope; honest limits: the README quick start targets the MiniCPM-o 4.5 full-duplex voice pipeline, 4.7 loading details are untested by this site and follow the official README). Route 3, the Ollama shortcut (the official library still lists only 4.6 as of October 15; the OpenBMB publisher page openbmb/minicpm-v4.7 runs today via ollama run - 1b about 1.8 GB, 35b-a3b Q4_K_M about 22 GB and the default latest tag, so small-memory machines must specify :1b explicitly; this route runs whole-model quants with no MoE-SSD, so "3 GB runs 35B" does not apply). One acceptance test for all three: ask about a local image and get a relevant answer. Division of labor with the same-batch open-source piece: that one covers capability and ecosystem, this one only covers getting it running. Repos: github.com/OpenBMB/MiniCPM-V and github.com/OpenBMB/MiniCPM-V-Apps.

Oct 15, 202610 min read
Field SOP

Nemotron 3.5 Bolt Local Deployment: 74B on a 24GB Card

NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on October 15, 2026: a compression of the 120B Super VL via Star Elastic budget-conditioned architecture search (expert top-K down from 22 to 9, distilled recovery training over 120B tokens), on a hybrid Mamba-2 and Transformer architecture. The quantization plan is the story - the expert layers that dominate the weights drop to IQ2_XXS at 2.0625 bits per element (about 19GB), Mamba linear layers use NVFP4, and lm_head, embeddings, and attention stay at Q8_0, landing the full model at 22-24GB with accuracy recovered through quantization-aware distillation from the compression stage onward. Four deployment routes, with commands: Ollama (supported since 0.40.5, ollama run nemotron-3.5-bolt, a 25GB download, verify 100 percent GPU via ollama ps); Unsloth (UD-IQ2_XXS about 22GB, UD-IQ1_M about 20.3GB, ctx from a conservative 4096); llama.cpp official recipes (builds after October 15 to recognize nemotron_h_moe; the 28GB recipe runs 65K context with 8-bit KV and 6 draft tokens, the 32GB recipe reaches 262K; llama-server defaults to port 9931 since b11521); and vLLM (Blackwell SM120 with the LiLiCorr drafter for multi-user serving). Two context starting points coexist: 16K-32K for bare weights, 4K for Unsloth whose default drafter eats another 1-2GB. Eight troubleshooting entries cover OOM context cuts, layers spilling to system RAM, the missing --mmproj flag, and the shared 16384 max_tokens budget. OpenMDW-1.1 grants free unrestricted commercial use, leaves outputs unencumbered, requires notices only on redistribution, and carries a patent-retaliation clause. Vendor-reported: GPQA Diamond 86.4, Terminal-Bench 2.1 at 51.0, PinchBench first in class (83.99 TPS on RTX Spark; no official consumer-card figure). The 4GB Bolt E4B followed on October 23.

Oct 23, 20268 min read