Open Source
Open Source

Announced GA, Yet Only 71 Stars: What Is Kinference?

Buoyant, the company behind Linkerd, open-sourced Kinference as an inference control plane for Kubernetes and vLLM and announced GA on November 4, 2026; a GitHub API snapshot on November 9, 2026 shows Kinference-AI/kinference (migrated from buoyantio/kinference via 301) at only 71 stars, 6 forks and 57 open issues, in Rust under Apache-2.0, created October 5, last pushed November 8, version v0.1.0-alpha.7. The README confesses verbatim that Kinference is alpha software with no sustained production exposure, lacking Anthropic-style /v1/messages, rate limiting, semantic classifiers and multi-replica support. Mechanically it exposes a gRPC interface that takes OpenAI-format requests, returns the chosen endpoint plus a mutated body, monitors vLLM KV-cache events and engine metrics, reuses NVIDIA's dynamo-kv-router crate and pairs with the Buoyant Gateway. Its vendor-reported median TTFT improvement, 6.88s to 0.31s on shared-prefix traffic, was measured against random routing rather than llm-d or Dynamo, and has no third-party reproduction.

Published November 10, 20269 min read
<!-- kinference-vllm-routing-opensource-resource | open-source | Announced GA, Yet Only 71 Stars: What Is Kinference? -->

November 2026 has an open-source project where the headlines and the repository tell two different stories. Buoyant, the company behind Linkerd, open-sourced Kinference, positioned as an inference control plane for Kubernetes and vLLM, and announced GA on November 4, 2026. Tech coverage followed quickly, with several outlets treating the launch as a major open-source event. Yet a GitHub API check pulled on November 9, 2026 shows the repository Kinference-AI/kinference at 71 stars, 6 forks and 57 open issues. The code is mostly Rust, the license is Apache-2.0, the repository was created on October 5, 2026, last pushed on November 8, 2026, and the current version tag is v0.1.0-alpha.7. A GA announcement and an alpha confession living on the same project is a gap worth unpacking, not mocking, and this article does the unpacking.

A quick note on how this batch divides the work, so nothing gets crossed. This article is a deep dive into a single repository, focused on the cluster-side routing decision layer: which endpoint should a request go to, and why KV-cache state should drive that decision. The Qoduck anonymous model hotspot breakdown covers that release event and the boundaries of its official wording; the free LLM API tier comparison compares what hosted free tiers give and take; the Yuanqi shutdown migration SOP is a damage-control manual for existing users. All three are different threads from this one. Within this site, the local LLM deployment comparison review sits on another layer entirely: that piece is about picking tools to run models on your own machine, while this piece is about the routing control plane in front of inference services that are already deployed as a cluster — the question is not where to run the model, but where to send the request. The Rust project roundup from November 9 also starts from a fresh Rust project, but a different object; this article does no roundups and dismantles only this one repository.

The Media Buzz and the 71-Star Reality

The first gap to address is between reporting heat and repository heat. The GA announcement came from Buoyant on November 4, 2026, and multiple media outlets covered it from the open-source-launch angle. Per the GitHub API data pulled on November 9, 2026, the repository Kinference-AI/kinference has 71 stars, 6 forks and 57 open issues. It was created on October 5, 2026, last pushed on November 8, 2026, and carries the tag v0.1.0-alpha.7. A repository barely a month old with a double-digit star count is not remarkable by itself; what deserves attention is the lag between coverage and adoption. Stories framed as a big open-source moment describe, on the code side, an early repository that has not seen wide uptake.

There is no mockery in that reading, because both halves can be true at once. What received the GA announcement is Buoyant's commercial product posture; what was open-sourced is alpha-stage code. When both land in the same news cycle, the impression of production readiness forms on its own, and the README pushes back against that impression directly, as the later sections show. One more logistics note belongs here: the repository moved. The old address buoyantio/kinference now returns a 301 redirect to the new organization, so if you bookmarked the old link at some point, update it.

What a Control Plane Does, and Why KV-Cache Matters

The README opens with one line: Kinference is an inference control plane for Kubernetes and vLLM. A control plane is not a load balancer, and this distinction is the starting point for understanding the whole project. As the README puts it: At its core, Kinference selects among inference endpoints, applying policies, routing, and load balancing as appropriate. Its core job is choosing among multiple inference endpoints, layering on policies, routing and load balancing where appropriate. The interface is spelled out as well: It exposes a gRPC interface that takes an inference request (e.g. in OpenAI chat-completions format) and returns the endpoint for that request, as well as the appropriate-mutated body. A request comes in, the control plane picks the endpoint and rewrites the body, and the caller sends the request onward.

Why go to that trouble? The problem statement, following the official framing, is that a plain load balancer does not know which machine holds your cache. LLM inference has a well-known property: the KV-cache. If a request lands on a replica that already holds its KV-cache, the prefill work can be reused; if it lands on a machine without that cache, the computation starts over. The more multi-turn the conversations and the more shared the prefixes, the larger the waste. Kinference's answer, verbatim from the README: Kinference monitors vLLM KV-cache events and engine metrics. It continuously listens to vLLM's KV-cache events and engine metrics, and that real-time state drives the per-request decision.

One implementation detail matters for how you read the project's trajectory. The README says Kinference currently uses NVIDIA's dynamo-kv-router crate to handle cache events. Cache awareness is not built from scratch; it leans on a crate NVIDIA already ships. That signals two things at once: mature components exist in the ecosystem, and part of the project's core capability moves at the pace of an upstream dependency. The positioning inside the topology is explicit too: Kinference is designed to be paired with a proxy. For example, it pairs with the Buoyant Gateway. The proxy faces the incoming traffic; the control plane makes the routing decisions. It is one component in a chain, not a whole that absorbs the rest of the chain.

Five Policy Classes and Five CRDs

State-aware routing also has to be rule-controlled, and that layer is expressed through Kubernetes CRDs and policies. Per the official documentation, Kinference defines five CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet, carrying the declarations for backends, policies, pools, providers and worker sets respectively. The policy catalog lists five classes: simple-to-cheaper, which prefers the cheaper path when possible; burst-to-cloud, which overflows peak traffic to cloud capacity; PII handling on local replicas, keeping sensitive data away from outside machines; batch processing with rate limiting; and model aliases, one name that resolves to different backends. All of this comes from the official documentation, and details beyond what this piece verified are deliberately left alone.

The tooling includes a kinfctl CLI. Per the official documentation, it validates policies before you send a request, so a miswritten rule surfaces before real traffic does, not after. Building the CLI runs through just install-cli:

sh
just install-cli

Counts and catalogs like these tend to move during an alpha phase. Before you build anything on them, treat the documentation inside the repository as the source of truth; this article reports the state as of November 9, 2026, and no later.

The Self-Reported Numbers, and Their Baseline

The performance story is the most quoted and the most context-hungry part of this launch. The official self-reported figures are these: in a shared-prefix scenario, median TTFT drops from 6.88s to 0.31s, a reduction of roughly 95 percent; without a shared prefix, 6.92s versus 6.83s, almost no gain. Two qualifiers have to travel with those numbers wherever they go. First, they are officially self-reported, with no third-party reproduction so far; until someone independent reruns them, they should be read as directional evidence, not as a capacity-planning input. Second, the control baseline is random routing, not llm-d, Dynamo or comparable routing projects. The numbers therefore measure cache-aware routing against blind routing, not Kinference against other cache-aware routers.

Credit where due: the official material itself states that the no-shared-prefix case shows almost no gain. That honesty helps you match the mechanism to your own traffic. If your requests are scattered and prefix-free short bursts, this mechanism has little to save in the first place; if your traffic is multi-turn chat or RAG built on a shared system prompt, shared prefixes concentrate, and that is exactly the scenario the mechanism targets. Match the scenario first, then quote the 95 percent figure.

GA and Alpha, Side by Side

The most counterintuitive fact about this project is that two status claims coexist. On November 4, 2026, Buoyant announced GA; the repository tag remains v0.1.0-alpha.7, and the Known limitations section of the README carries a blunt confession, verbatim: Kinference is alpha software. It has not seen sustained production exposure. Run at your own risk.

The same section lists four known gaps: no /v1/messages, the Anthropic-style endpoint; no rate limiting; no semantic classifier; no multi-replica support. The README says these will be addressed shortly, without promising dates. The reasonable reading is that the GA announcement covers Buoyant's product posture, including the paired Buoyant Gateway side, while the open-source repository itself is early code, and neither half backs the other. This article therefore avoids phrases like production-ready or a drop-in replacement for your existing gateway, and you should keep the same restraint when reading: until the project's own materials claim production exposure, treat it as a repository to read, to test and to file issues against.

License, Stack and Ecosystem

The license is Apache-2.0, with the LICENSE file carrying the copyright line Copyright 2026 Buoyant, Inc. The primary language is Rust. Apache-2.0 is a permissive license with an explicit patent grant; whether it fits your organization's compliance process is a question for the license text and your counsel, not for this article, which offers no legal advice.

The ecosystem dependency deserves its own paragraph. Handling KV-cache events goes through NVIDIA's dynamo-kv-router crate, which means the project did not rebuild cache awareness from zero. The upside is faster implementation aligned with a mature component; the cost is that this core capability partly tracks an upstream dependency. Add the stated pairing with the Buoyant Gateway, and the overall shape becomes clear: a small, pragmatic team open-sourcing the decision layer, while traffic ingress, engine metrics and cache events stay with dedicated components around it.

Who Should Try It Now

Two groups fit a closer look right now. The first runs vLLM clusters with concentrated shared-prefix traffic — multi-turn assistants, RAG services built on a common system prompt — and wants to study how KV-cache-aware routing is engineered. Kinference is a small, readable Rust codebase, and the path from event listening to gRPC decisions is complete enough to learn from. The second is teams willing to deploy outside production, to test while filing issues, and to wait out the four known gaps one by one.

The groups that should not bet on it now are equally clear: anyone expecting a drop-in replacement for a production gateway, needing multi-replica support and rate limiting out of the box, or planning to fold the self-reported 95 percent figure directly into capacity planning. For those needs, wait until the limitation list shortens and third-party numbers appear. The way in is simple: clone the repository at the plain address https://github.com/Kinference-AI/kinference (migrated from the old buoyantio/kinference address, which now 301-redirects), build kinfctl with just install-cli, validate your policies before sending traffic, and read the Known limitations section before anything else.

FAQ

Q1: What does Kinference actually do?

A1: It is an inference control plane open-sourced by Buoyant, running on Kubernetes alongside vLLM. Per the README, it monitors vLLM KV-cache events and engine metrics, exposes a gRPC interface that takes an inference request in OpenAI chat-completions format, and returns the endpoint for that request plus the appropriately mutated body. In one sentence: it decides which replica a request goes to, and a proxy such as the Buoyant Gateway pairs with it to face the traffic.

Q2: How can a project announced GA on November 4 still be alpha?

A2: Because the two claims sit at different layers. The GA announcement is Buoyant's product-level statement dated November 4, 2026, while the repository tag remains v0.1.0-alpha.7 and the README states verbatim: Kinference is alpha software. It has not seen sustained production exposure. Run at your own risk. The reasonable reading is that GA covers the commercial product posture while the open-source repository is early code; GA is not a maturity certificate for the repository.

Q3: Can I trust the reported 95 percent TTFT reduction?

A3: Not without qualifiers. The numbers are officially self-reported, with no third-party reproduction yet: median TTFT in a shared-prefix scenario drops from 6.88s to 0.31s, while without a shared prefix 6.92s versus 6.83s shows almost no gain. The control baseline is random routing, not llm-d, Dynamo or comparable projects, so the figures measure cache-aware routing against blind routing. Treat them as directional until an independent party reruns the benchmark.

Q4: Can I put it in production and replace my existing gateway?

A4: Not recommended, and that is not its stated role. The README confesses alpha software without sustained production exposure, and Known limitations lists four gaps: no /v1/messages Anthropic-style endpoint, no rate limiting, no semantic classifier, no multi-replica support. The README also says Kinference is designed to pair with a proxy, for example the Buoyant Gateway, so the relationship is a pairing, not a replacement.

Q5: What is the license, the language, and how do I start?

A5: The license is Apache-2.0, with the copyright line Copyright 2026 Buoyant, Inc., and the primary language is Rust. To start, clone the repository — note that the old address buoyantio/kinference 301-redirects to Kinference-AI/kinference — build the kinfctl CLI with just install-cli, validate your policies before sending requests, and read the Known limitations section. The five CRDs and five policy classes can be checked against the official documentation.

Related reading

This article is AI-assisted and human-edited. Last updated: 2026-11-10

FAQ

What does Kinference actually do?
It is an inference control plane open-sourced by Buoyant, running on Kubernetes alongside vLLM. Per the README, it monitors vLLM KV-cache events and engine metrics, exposes a gRPC interface that takes an inference request in OpenAI chat-completions format, and returns the endpoint for that request plus the appropriately mutated body. In one sentence: it decides which replica a request goes to, and a proxy such as the Buoyant Gateway pairs with it to face the traffic.
How can a project announced GA on November 4 still be alpha?
Because the two claims sit at different layers. The GA announcement is Buoyant's product-level statement dated November 4, 2026, while the repository tag remains v0.1.0-alpha.7 and the README states verbatim: Kinference is alpha software. It has not seen sustained production exposure. Run at your own risk. The reasonable reading is that GA covers the commercial product posture while the open-source repository is early code; GA is not a maturity certificate for the repository.
Can I trust the reported 95 percent TTFT reduction?
Not without qualifiers. The numbers are officially self-reported, with no third-party reproduction yet: median TTFT in a shared-prefix scenario drops from 6.88s to 0.31s, while without a shared prefix 6.92s versus 6.83s shows almost no gain. The control baseline is random routing, not llm-d, Dynamo or comparable projects, so the figures measure cache-aware routing against blind routing. Treat them as directional until an independent party reruns the benchmark.
Can I put it in production and replace my existing gateway?
Not recommended, and that is not its stated role. The README confesses alpha software without sustained production exposure, and Known limitations lists four gaps: no /v1/messages Anthropic-style endpoint, no rate limiting, no semantic classifier, no multi-replica support. The README also says Kinference is designed to pair with a proxy, for example the Buoyant Gateway, so the relationship is a pairing, not a replacement.
What is the license, the language, and how do I start?
The license is Apache-2.0, with the copyright line Copyright 2026 Buoyant, Inc., and the primary language is Rust. To start, clone the repository — note that the old address buoyantio/kinference 301-redirects to Kinference-AI/kinference — build the kinfctl CLI with just install-cli, validate your policies before sending requests, and read the Known limitations section. The five CRDs and five policy classes can be checked against the official documentation.

Related

Open Source

3,003 Stars, No Stable Release Yet: What Is Blocking It?

oxc-project/tsrs is a Rust rewrite of the TypeScript 7 compiler and language server, described in its README as a drop-in replacement. A GitHub API snapshot on November 8, 2026 showed 3,003 stars and Apache-2.0, but still no stable release. The README says the team aims to ship a stable version against TypeScript 7.1 with 100% conformance; aim is a target, not a delivered result. docs/STATUS.md lists GitHub prereleases, a manually assigned npm beta tag and release candidates, but nowhere declares stable. Speedup figures come from three unrelated setups that cannot be merged into one table: the project's own bench/RESULTS.md covers 19 projects on an Apple M3 Max, measured 2026-10-28 with Hyperfine medians against TypeScript 7.1.0-beta, giving 1.71x-6.62x faster; Cloudflare's up to 8x compares against TypeScript 7.0; third-party numbers near 2.7x and 5.3x have their own machines and baselines. This is a deep dive into one repository, not a multi-project ranking.

Nov 9, 20269 min read
Open Source

19,207 stars, still alpha: can this Rust engine rebuild a site?

storytold/craft-anything describes itself as an open-source clean-room engine in pure Rust: see how any website, app or game works, then have AI build your own version. The workflow points at a target, states what is wanted, studies the original with real inspection tools, has a coding agent build a new project, compares results against the original and iterates. A November 7, 2026 GitHub API check showed 19,207 stars, Rust as the language and a repository created on October 31, 2026. The README badges say early alpha and 100% Rust, and the license text is dual-licensed under MIT or Apache-2.0 at your option. ATTRIBUTION.md records bundled font and icon licenses while brand assets must not be modified. macOS, Windows, Linux, FreeBSD and Web are supported through a desktop app, CLI and MCP server, with Anthropic, OpenAI, OpenRouter, xAI and local models plus installed GitHub Copilot, ACP grok and Ghidra. The README admits some capabilities are built but not yet run live.

Nov 7, 20269 min read
Open Source

AWS Infinity: an Agent Runtime Where Waiting Costs Nothing

AWS open-sourced Infinity, a suspend-and-resume agent runtime: the code lives in the hydro-project organization (aws/ and aws/hydro both 404), Apache-2.0, written in Rust, from the AWS x UC Berkeley Hydro collaboration. The official benchmark runs 80,000 agents for 20 rounds each — 3.2 million completions — at roughly 100KB of memory per agent on a single-threaded EPYC 9R14, extrapolating to about 75,000 agents in 8GB (an extrapolation; the Raspberry Pi 5 is only a memory analogy). This deep dive covers the serverless four-piece design, the simulated-model qualifier, the community critique list (durable I/O, race transfer, 25 stars and untested by the market), and how to get started.

Oct 26, 202611 min read