November 2026 has an open-source project where the headlines and the repository tell two different stories. Buoyant, the company behind Linkerd, open-sourced Kinference, positioned as an inference control plane for Kubernetes and vLLM, and announced GA on November 4, 2026. Tech coverage followed quickly, with several outlets treating the launch as a major open-source event. Yet a GitHub API check pulled on November 9, 2026 shows the repository Kinference-AI/kinference at 71 stars, 6 forks and 57 open issues. The code is mostly Rust, the license is Apache-2.0, the repository was created on October 5, 2026, last pushed on November 8, 2026, and the current version tag is v0.1.0-alpha.7. A GA announcement and an alpha confession living on the same project is a gap worth unpacking, not mocking, and this article does the unpacking.
A quick note on how this batch divides the work, so nothing gets crossed. This article is a deep dive into a single repository, focused on the cluster-side routing decision layer: which endpoint should a request go to, and why KV-cache state should drive that decision. The Qoduck anonymous model hotspot breakdown covers that release event and the boundaries of its official wording; the free LLM API tier comparison compares what hosted free tiers give and take; the Yuanqi shutdown migration SOP is a damage-control manual for existing users. All three are different threads from this one. Within this site, the local LLM deployment comparison review sits on another layer entirely: that piece is about picking tools to run models on your own machine, while this piece is about the routing control plane in front of inference services that are already deployed as a cluster — the question is not where to run the model, but where to send the request. The Rust project roundup from November 9 also starts from a fresh Rust project, but a different object; this article does no roundups and dismantles only this one repository.
The Media Buzz and the 71-Star Reality
The first gap to address is between reporting heat and repository heat. The GA announcement came from Buoyant on November 4, 2026, and multiple media outlets covered it from the open-source-launch angle. Per the GitHub API data pulled on November 9, 2026, the repository Kinference-AI/kinference has 71 stars, 6 forks and 57 open issues. It was created on October 5, 2026, last pushed on November 8, 2026, and carries the tag v0.1.0-alpha.7. A repository barely a month old with a double-digit star count is not remarkable by itself; what deserves attention is the lag between coverage and adoption. Stories framed as a big open-source moment describe, on the code side, an early repository that has not seen wide uptake.
There is no mockery in that reading, because both halves can be true at once. What received the GA announcement is Buoyant's commercial product posture; what was open-sourced is alpha-stage code. When both land in the same news cycle, the impression of production readiness forms on its own, and the README pushes back against that impression directly, as the later sections show. One more logistics note belongs here: the repository moved. The old address buoyantio/kinference now returns a 301 redirect to the new organization, so if you bookmarked the old link at some point, update it.
What a Control Plane Does, and Why KV-Cache Matters
The README opens with one line: Kinference is an inference control plane for Kubernetes and vLLM. A control plane is not a load balancer, and this distinction is the starting point for understanding the whole project. As the README puts it: At its core, Kinference selects among inference endpoints, applying policies, routing, and load balancing as appropriate. Its core job is choosing among multiple inference endpoints, layering on policies, routing and load balancing where appropriate. The interface is spelled out as well: It exposes a gRPC interface that takes an inference request (e.g. in OpenAI chat-completions format) and returns the endpoint for that request, as well as the appropriate-mutated body. A request comes in, the control plane picks the endpoint and rewrites the body, and the caller sends the request onward.
Why go to that trouble? The problem statement, following the official framing, is that a plain load balancer does not know which machine holds your cache. LLM inference has a well-known property: the KV-cache. If a request lands on a replica that already holds its KV-cache, the prefill work can be reused; if it lands on a machine without that cache, the computation starts over. The more multi-turn the conversations and the more shared the prefixes, the larger the waste. Kinference's answer, verbatim from the README: Kinference monitors vLLM KV-cache events and engine metrics. It continuously listens to vLLM's KV-cache events and engine metrics, and that real-time state drives the per-request decision.
One implementation detail matters for how you read the project's trajectory. The README says Kinference currently uses NVIDIA's dynamo-kv-router crate to handle cache events. Cache awareness is not built from scratch; it leans on a crate NVIDIA already ships. That signals two things at once: mature components exist in the ecosystem, and part of the project's core capability moves at the pace of an upstream dependency. The positioning inside the topology is explicit too: Kinference is designed to be paired with a proxy. For example, it pairs with the Buoyant Gateway. The proxy faces the incoming traffic; the control plane makes the routing decisions. It is one component in a chain, not a whole that absorbs the rest of the chain.
Five Policy Classes and Five CRDs
State-aware routing also has to be rule-controlled, and that layer is expressed through Kubernetes CRDs and policies. Per the official documentation, Kinference defines five CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet, carrying the declarations for backends, policies, pools, providers and worker sets respectively. The policy catalog lists five classes: simple-to-cheaper, which prefers the cheaper path when possible; burst-to-cloud, which overflows peak traffic to cloud capacity; PII handling on local replicas, keeping sensitive data away from outside machines; batch processing with rate limiting; and model aliases, one name that resolves to different backends. All of this comes from the official documentation, and details beyond what this piece verified are deliberately left alone.
The tooling includes a kinfctl CLI. Per the official documentation, it validates policies before you send a request, so a miswritten rule surfaces before real traffic does, not after. Building the CLI runs through just install-cli:
just install-cliCounts and catalogs like these tend to move during an alpha phase. Before you build anything on them, treat the documentation inside the repository as the source of truth; this article reports the state as of November 9, 2026, and no later.
The Self-Reported Numbers, and Their Baseline
The performance story is the most quoted and the most context-hungry part of this launch. The official self-reported figures are these: in a shared-prefix scenario, median TTFT drops from 6.88s to 0.31s, a reduction of roughly 95 percent; without a shared prefix, 6.92s versus 6.83s, almost no gain. Two qualifiers have to travel with those numbers wherever they go. First, they are officially self-reported, with no third-party reproduction so far; until someone independent reruns them, they should be read as directional evidence, not as a capacity-planning input. Second, the control baseline is random routing, not llm-d, Dynamo or comparable routing projects. The numbers therefore measure cache-aware routing against blind routing, not Kinference against other cache-aware routers.
Credit where due: the official material itself states that the no-shared-prefix case shows almost no gain. That honesty helps you match the mechanism to your own traffic. If your requests are scattered and prefix-free short bursts, this mechanism has little to save in the first place; if your traffic is multi-turn chat or RAG built on a shared system prompt, shared prefixes concentrate, and that is exactly the scenario the mechanism targets. Match the scenario first, then quote the 95 percent figure.
GA and Alpha, Side by Side
The most counterintuitive fact about this project is that two status claims coexist. On November 4, 2026, Buoyant announced GA; the repository tag remains v0.1.0-alpha.7, and the Known limitations section of the README carries a blunt confession, verbatim: Kinference is alpha software. It has not seen sustained production exposure. Run at your own risk.
The same section lists four known gaps: no /v1/messages, the Anthropic-style endpoint; no rate limiting; no semantic classifier; no multi-replica support. The README says these will be addressed shortly, without promising dates. The reasonable reading is that the GA announcement covers Buoyant's product posture, including the paired Buoyant Gateway side, while the open-source repository itself is early code, and neither half backs the other. This article therefore avoids phrases like production-ready or a drop-in replacement for your existing gateway, and you should keep the same restraint when reading: until the project's own materials claim production exposure, treat it as a repository to read, to test and to file issues against.
License, Stack and Ecosystem
The license is Apache-2.0, with the LICENSE file carrying the copyright line Copyright 2026 Buoyant, Inc. The primary language is Rust. Apache-2.0 is a permissive license with an explicit patent grant; whether it fits your organization's compliance process is a question for the license text and your counsel, not for this article, which offers no legal advice.
The ecosystem dependency deserves its own paragraph. Handling KV-cache events goes through NVIDIA's dynamo-kv-router crate, which means the project did not rebuild cache awareness from zero. The upside is faster implementation aligned with a mature component; the cost is that this core capability partly tracks an upstream dependency. Add the stated pairing with the Buoyant Gateway, and the overall shape becomes clear: a small, pragmatic team open-sourcing the decision layer, while traffic ingress, engine metrics and cache events stay with dedicated components around it.
Who Should Try It Now
Two groups fit a closer look right now. The first runs vLLM clusters with concentrated shared-prefix traffic — multi-turn assistants, RAG services built on a common system prompt — and wants to study how KV-cache-aware routing is engineered. Kinference is a small, readable Rust codebase, and the path from event listening to gRPC decisions is complete enough to learn from. The second is teams willing to deploy outside production, to test while filing issues, and to wait out the four known gaps one by one.
The groups that should not bet on it now are equally clear: anyone expecting a drop-in replacement for a production gateway, needing multi-replica support and rate limiting out of the box, or planning to fold the self-reported 95 percent figure directly into capacity planning. For those needs, wait until the limitation list shortens and third-party numbers appear. The way in is simple: clone the repository at the plain address https://github.com/Kinference-AI/kinference (migrated from the old buoyantio/kinference address, which now 301-redirects), build kinfctl with just install-cli, validate your policies before sending traffic, and read the Known limitations section before anything else.
FAQ
Q1: What does Kinference actually do?
A1: It is an inference control plane open-sourced by Buoyant, running on Kubernetes alongside vLLM. Per the README, it monitors vLLM KV-cache events and engine metrics, exposes a gRPC interface that takes an inference request in OpenAI chat-completions format, and returns the endpoint for that request plus the appropriately mutated body. In one sentence: it decides which replica a request goes to, and a proxy such as the Buoyant Gateway pairs with it to face the traffic.
Q2: How can a project announced GA on November 4 still be alpha?
A2: Because the two claims sit at different layers. The GA announcement is Buoyant's product-level statement dated November 4, 2026, while the repository tag remains v0.1.0-alpha.7 and the README states verbatim: Kinference is alpha software. It has not seen sustained production exposure. Run at your own risk. The reasonable reading is that GA covers the commercial product posture while the open-source repository is early code; GA is not a maturity certificate for the repository.
Q3: Can I trust the reported 95 percent TTFT reduction?
A3: Not without qualifiers. The numbers are officially self-reported, with no third-party reproduction yet: median TTFT in a shared-prefix scenario drops from 6.88s to 0.31s, while without a shared prefix 6.92s versus 6.83s shows almost no gain. The control baseline is random routing, not llm-d, Dynamo or comparable projects, so the figures measure cache-aware routing against blind routing. Treat them as directional until an independent party reruns the benchmark.
Q4: Can I put it in production and replace my existing gateway?
A4: Not recommended, and that is not its stated role. The README confesses alpha software without sustained production exposure, and Known limitations lists four gaps: no /v1/messages Anthropic-style endpoint, no rate limiting, no semantic classifier, no multi-replica support. The README also says Kinference is designed to pair with a proxy, for example the Buoyant Gateway, so the relationship is a pairing, not a replacement.
Q5: What is the license, the language, and how do I start?
A5: The license is Apache-2.0, with the copyright line Copyright 2026 Buoyant, Inc., and the primary language is Rust. To start, clone the repository — note that the old address buoyantio/kinference 301-redirects to Kinference-AI/kinference — build the kinfctl CLI with just install-cli, validate your policies before sending requests, and read the Known limitations section. The five CRDs and five policy classes can be checked against the official documentation.
Related reading