High-quality vs economical indexing-how to choose?

Few docs (<10) and questions phrased nearly verbatim to the source: economical saves money. Every other production scenario: high-quality, always. The test: if a user might ask "annual leave" when the doc says "paid time off," you need semantic retrieval.

Dify defaults to Weaviate, ready out of the box. As you scale, pick by infrastructure: already on Postgres use pgvector, on Kubernetes use Qdrant or Milvus, small teams do fine with Chroma. Switching stores forces a reindex-decide before building.

How to debug inaccurate retrieval?

Three steps: first disable the threshold (set 0) and check whether raw top-k hits the target fragment-no hit means a recall-stage problem (check embedding choice, chunk size); hit but wrong answer means a ranking-stage problem (enable rerank); hit and ranked right but still wrong means a prompt or LLM-capability problem.

Can the enterprise intranet use local models?

Yes. Dify self-hosted + Xinference / Ollama for local LLM and embedding (e.g., qwen2.5 + bge-m3) keeps the entire chain inside the intranet, meeting strong-compliance scenarios like finance and healthcare.

Build an Enterprise Knowledge Base RAG with Dify SOP

Q: High-quality vs economical indexing-how to choose?

Few docs (<10) and questions phrased nearly verbatim to the source: economical saves money. Every other production scenario: high-quality, always. The test: if a user might ask "annual leave" when the doc says "paid time off," you need semantic retrieval.

Q: Which vector store?

Dify defaults to Weaviate, ready out of the box. As you scale, pick by infrastructure: already on Postgres use pgvector, on Kubernetes use Qdrant or Milvus, small teams do fine with Chroma. Switching stores forces a reindex-decide before building.

Q: How to debug inaccurate retrieval?

Three steps: first disable the threshold (set 0) and check whether raw top-k hits the target fragment-no hit means a recall-stage problem (check embedding choice, chunk size); hit but wrong answer means a ranking-stage problem (enable rerank); hit and ranked right but still wrong means a prompt or LLM-capability problem.

Q: Reindex after doc updates?

Yes. For single-doc changes, use Dify's "Reindex" button; for batch updates, an API script. Otherwise RAG keeps answering from stale content.

Q: Can the enterprise intranet use local models?

Yes. Dify self-hosted + Xinference / Ollama for local LLM and embedding (e.g., qwen2.5 + bge-m3) keeps the entire chain inside the intranet, meeting strong-compliance scenarios like finance and healthcare.

Stuffing dozens of company policy docs, product manuals, and contract templates into ChatGPT gets you either hallucinated answers or a token bill that hurts. Rolling your own RAG pipeline with LangChain means hand-wiring chunking, embedder selection, vector store, retrieval logic, and citation tracing-each step more engineering than the business logic itself, and one tuning tweak forces you to retrace half the project. Teams who've been down this road all learn the same lesson: the hard part of RAG isn't getting it to run, it's keeping retrieval quality stable.

This piece skips concepts and walks the full flow of self-hosting Dify to build an enterprise KB: deployment, KB creation, chunking, embedding selection, retrieval tuning, app assembly. Each step ships real params and copy-paste prompts, with pitfalls and an FAQ at the end. Compared to the AnythingLLM (ready-to-run local RAG) and Langflow (visual workflow platform) covered on this site, Dify is "a self-hostable LLM app dev platform"-the KB is one capability, but its RAG tuning granularity beats AnythingLLM and its onboarding is gentler than Langflow. The fit for production-grade enterprise KBs.

1. Why Dify: Positioning and Boundaries

Nail down Dify's place first. AnythingLLM is a "install and you've got a UI, drag docs and chat" finished app-winning on full-stack local and out-of-the-box, but short on retrieval knobs, for individuals and small teams going fast. Langflow is a visual workflow platform-drag components to build chains, flexible but onboarding demands understanding nodes and orchestration, leaning more "build flows" than "build KBs." Dify lands in between: both a ready-to-use app platform (built-in chatbot, Agent, workflow templates) and a system exposing every RAG stage as a tunable knob-chunking, embedding, retrieval mode, rerank, top-k, score threshold all twistable in the UI.

The clincher: Dify is open-source and self-hostable, so data never leaves your domain, and runs on enterprise intranets. Legal contracts, internal code, customer files never touch any third-party SaaS. Everything below is based on the self-hosted Community Edition.

2. Self-Host Dify: Docker Compose in One Shot

Dify's official path is Docker Compose, one command spinning up the whole stack (Web, API, Worker, vector store, Redis, Postgres).

bash

# 1. Clone the repo
git clone https://github.com/langgenius/dify.git
cd dify/docker

# 2. Copy env vars
cp .env.example .env

# 3. Edit .env as needed (ports, secrets, vector store)
#    Default vector store Weaviate, web port 80

# 4. Bring up all services
docker compose up -d

# 5. Check status
docker compose ps

After startup, open http://localhost (or server IP); first entry sets the admin account and password. Dify is running, but one critical step remains: model providers.

Go to "Settings > Model Providers" and configure at least two model types:

LLM: closed-source-fill API keys for OpenAI / Anthropic / DeepSeek; intranet-bring local models in via Xinference or Ollama (e.g., qwen2.5, deepseek-r1).
Embedding: needed to vectorize KB docs-selection gets its own section below. Wire the provider in first so you can pick the model at KB creation.

Pitfall: Docker uses port 80 by default; if taken, change EXPOSE_NGINX_PORT in .env. For production, always change SECRET_KEY and enable HTTPS-never ship the default secret. Vector store and embedding both eat RAM; start at 8GB + 4 cores, add as doc volume grows.

3. Build Your First Knowledge Base: Upload and Indexing Mode

Go to "Knowledge > Create Knowledge" and drag docs in. Dify supports PDF / Word / Excel / PPT / Markdown / TXT / HTML, plus pasting text or feeding a URL to crawl. After upload, the first choice is indexing mode-the first fork:

High-Quality indexing: an embedding model vectorizes docs into a vector store, supporting semantic and hybrid retrieval. Production uses only this, but it consumes embedding calls and vector store storage.
Economical indexing: keyword inverted index only, no vectorization-free, but full-text search only, near-zero semantic recall.

Many teams pick economical to save embedding costs, then ask "how much annual leave" and fail to retrieve the "paid time off" doc-keywords don't match. Hard rule: more than a few dozen docs or varied question phrasing, always high-quality. Economical is only for minimal scenarios with few docs and questions mirroring source text.

After indexing mode, the next step is chunking. This caps your retrieval quality ceiling, so it gets its own section.

4. Chunking Strategy: How to Set chunk size and overlap

Dify's segmentation has two modes: Automatic and Custom. Automatic splits by document structure (paragraphs, line breaks)-handy but inflexible; production should use Custom and keep the knobs in hand.

Core Custom params (all marked "tune to your corpus"):

Chunk length (chunk size): 500-1000 tokens recommended, tune to your corpus. Short docs (FAQs, SOP entries) 300-500; long docs (manuals, contract clauses) 800-1200.
Chunk overlap: 50-100 recommended, ~10% of chunk size, tune to your corpus. Overlap ensures boundary content isn't lost to a hard cut.
Separator: lead with structural markers (\n\n paragraphs, ### headings), then fall back to length. Don't hard-cut by fixed length only-it slices a complete clause in two, leaving both halves incomplete.

Two advanced modes worth calling out:

Parent-Child Chunking: retrieve with small chunks (child) for precision, feed the large chunk (parent) to the LLM for full context. Great for long docs needing both precise recall and complete answers.
Q&A segmentation: have the model extract "question-answer" pairs, then do semantic retrieval against the questions. Ideal for FAQ corpora-sharply improving question-doc matching, because you retrieve aligned questions, not raw text fragments.

A common myth: bigger chunks mean more complete context, so bigger is better. Wrong. Oversized chunks cram multiple topics into one fragment, diluting the embedding and lowering similarity to relevant questions; they also eat LLM context, reducing effective fragments that fit. Undersized chunks fracture context, yielding scattered answers. 500-1000 is a solid start-test on your own corpus. No silver bullet.

5. Embedding Selection: Don't Force English Models on Chinese Corpus

Embedding is RAG's foundation-pick wrong and your retrieval ceiling is locked. Dify supports many providers; selection comes down to three things: language, dimension, deployment.

Language match is the first principle. Chinese corpus needs a Chinese-optimized model, English an English model, multilingual a multilingual model. Common picks:

Chinese-first: bge-large-zh-v1.5 (open-source from BAAI, local via Xinference / Ollama, free) or OpenAI text-embedding-3-small (cloud, cheap).
Mixed Chinese-English / multilingual: bge-m3 (multilingual, sparse + dense retrieval, local free) or jina-embeddings-v3.
English-first: OpenAI text-embedding-3-large or Cohere embed-english-v3.0.

Dimension affects storage and speed. Higher dims = more expressiveness but bigger store and slower retrieval. 1024 is the common balance-bge-m3 is 1024-dim, OpenAI v3 offers 256/512/3072.

Deployment follows data sensitivity. Intranet, data-stays-in-domain: run bge via Xinference or Ollama; cloud is fine: OpenAI saves ops. Mixing is fine-but one KB binds to one embedding model, set at creation; swapping mid-way forces a full reindex.

Pitfall: don't force an English model on Chinese. OpenAI's embedding runs on Chinese, but semantic recall trails bge-large-zh noticeably-especially on synonyms and colloquial phrasing. Lock in your embedding before building the KB; it's a one-way decision.

6. Retrieval Tuning: top-k, Hybrid Search, and Rerank

A built KB is just the start; retrieval tuning is RAG's quality watershed. Dify's "Retrieval Settings" gives a set of knobs-let's go through each.

Three retrieval modes:

Vector retrieval (semantic): embedding similarity recall-strong on semantic match, weak on exact keywords. Ask "annual leave," retrieve "paid time off."
Full-text retrieval (keyword): BM25-style keyword matching-strong on exact terms, proper nouns, IDs; weak on semantics.
Hybrid retrieval: runs both, fuses by weight. Production should use this-semantic and keyword complement, steadiest recall.

top-k: how many fragments to feed the LLM, 3-5 recommended, tune to your corpus. Too small, miss recalls; too large, dilute context, waste tokens, risk feeding irrelevant fragments that induce hallucination. Start at 3, add if recall is short, but don't exceed 5-marginal returns drop sharply while cost and noise scale linearly.

Score threshold (similarity threshold): filters fragments below the threshold, ~0.5 to start, tune to your corpus. Too strict drops valid recalls; too loose floods noise. First disable the threshold (set 0) to inspect raw top-k quality, then set it in reverse.

Hybrid retrieval weight: semantic vs keyword, start 0.7 / 0.3, tune to your corpus. Raise semantic for fuzzy questions, raise keyword for exact-term queries.

Rerank model: the most overlooked yet highest-payoff step. After hybrid retrieval recalls top-k x N candidates, a rerank model re-scores and reorders, keeping only the most relevant few. Rerank isn't an embedding-it's a dedicated cross-encoder, far more precise than vector similarity. Dify supports Cohere Rerank (cloud, multilingual rerank-multilingual-v3.0), bge-reranker-v2-m3 (local, free), Jina Rerank.

Enable rerank in production, even at the cost of one extra model call. Without it, jacking top-k to lift recall just raises noise in lockstep, dilutes context, pushes the LLM off course. With rerank, dial top-k down (e.g., recall 5, take top 2 after rerank)-covering precision and cost.

7. App Assembly: Hang the KB on a Chatbot

The KB is tuned; now make it queryable. Among Dify's app types, "Chatbot" fits KB Q&A best.

Go to "Studio > Create Blank App > Chatbot," and in the orchestration page do three things:

1. Attach the KB. In "Context," add the KB and set the recall fragment count (matching retrieval top-k).

2. Write the System Prompt. This controls answer style and boundaries-here's a copy-paste template:

Prompt

You are {Company}'s internal knowledge assistant. Answer user questions only based on content retrieved from the knowledge base.

Rules:
1. Answer primarily using KB fragments, and cite source numbers at the end of sentences, e.g., "Annual leave is 10 days [1]."
2. If the KB has no relevant match, reply "No relevant information in the knowledge base; please contact HR / IT." Never fabricate or use the model's own knowledge to answer.
3. Answer in concise English, bullet points where possible; lead with the conclusion, then expand.
4. For sensitive info-amounts, policies, processes, contract clauses-quote the source wording verbatim; never rewrite or infer.
5. If the user's question is ambiguous, ask a clarifying question before retrieving.

Tone: professional, restrained, not verbose.

3. Test. Use "Debug & Preview" to validate via direct chat. Test two question types: ones with clear KB answers-check citation accuracy and source numbers; and ones the KB can't answer-check whether the model honestly declines or invents. If results fall short, go back and tune retrieval params or the prompt, then iterate.

Once tests pass, "Publish" as a Web App, or use Dify's API to wire it into your system. Dify can also embed the assistant into web pages or enterprise IM.

8. Six Pitfalls from the Trenches

1. Economical indexing saves pennies, wrecks retrieval. Picking economical to dodge embedding costs means keyword search misses synonyms, recall collapses. Production always uses high-quality-the embedding cost is a rounding error next to wrong answers.

2. English embedding on Chinese corpus. Running an English model on Chinese trails a Chinese model by a notch, especially on colloquial questions and synonyms. Lead with language match-bge-large-zh / bge-m3 is the safe Chinese default.

3. Oversized chunks dilute recall. Setting chunk to 2000+ for "context completeness" crams multiple topics into one fragment, diluting the embedding and dropping similarity to relevant questions. Start at 500-1000 and test on your corpus.

4. No rerank, just cranking top-k. Thinking more recall is better, you push top-k to 10-noise rises in lockstep, context dilutes, tokens burn. The right move is enable rerank and keep top-k at 3-5.

5. Forgot to reindex after doc updates. A policy doc in the KB changed but didn't trigger reindex, so RAG keeps answering from the old version. Dify supports per-document reindex-trigger it after any change; for bulk updates, use an API script to batch rebuild.

6. Score threshold set too strict. A threshold above 0.7 filters out most valid recalls, and the model declines constantly. First use a 0 threshold to inspect the raw recall distribution, then set the threshold-don't eyeball it.

References