On September 22, while shipping MiMo-V2.6, Xiaomi quietly did something most coverage ignored: it forked verl. Within days that fork (XiaomiMiMo/verl) collected 465 stars, while its upstream, verl-project/verl, sat at 23,650 stars at the top of the open-source RL post-training chart. This was not a random fork. Put TRL, OpenRLHF, AReaL, and NeMo-RL on the same table and a clear picture emerges: the infrastructure for Agentic RL — training agents on verifiable task rewards — has grown into five visibly diverging engineering tracks. By Xiaomi's official figures, MiMo-V2.6 went through roughly six days of live RL over about 750,000 trajectories, and the framework-selection question behind numbers like that is now unavoidable for any team training its own agents.
The confusion is real: all five are Apache-2.0, all are Python, every README claims high performance and scalability — yet star counts span 2,033 to 23,650. If you look only at stars, you will conclude "just pick verl." But a large share of that gap is simply a head-start differential (TRL was created in March 2020, NeMo-RL in March 2025 — a five-year spread) plus differences in community positioning, not a clean ranking of capability.
First, the scope statement. All star counts, fork counts, licenses, creation dates, and last-push dates in this article come from a GitHub API snapshot taken on 2026-09-27. Performance numbers self-reported in each project's README are not used here — the evaluation harnesses differ, so benchmarks are not comparable across projects. What this article compares is positioning, ecosystem, engineering shape, and fit. A note on how this piece divides labor with existing coverage on this site: our open-source flagship model comparison evaluates the models themselves, while this piece evaluates training frameworks; our LLM fine-tuning SOP covers the general SFT/LoRA route and complements this framework comparison. For Xiaomi-side launch details see our MiMo-V2.6 release coverage; for a close reading of the fork itself see our Xiaomi verl fork resource. Neither is repeated here.
1. The Five Contenders at a Glance
The raw specs first (snapshot 2026-09-27; licenses as reported by the GitHub API license field):
| Framework | Repository | Stars | Forks | Created | License | Official positioning |
|---|---|---|---|---|---|---|
| verl | verl-project/verl | 23,650 | n/a | not sampled | Apache-2.0 | HybridFlow: a flexible and efficient RL post-training framework |
| TRL | huggingface/trl | 19,398 | 3,021 | 2020-03-27 | Apache-2.0 | Train transformer language models with reinforcement learning |
| OpenRLHF | OpenRLHF/OpenRLHF | 10,045 | 1,026 | 2023-07-30 | Apache-2.0 | An easy-to-use, scalable, high-performance Agentic RL framework based on Ray (PPO/DAPO/REINFORCE++) |
| AReaL | areal-project/AReaL | 5,796 | 612 | 2025-02-24 | Apache-2.0 | The RL bridge for LLM-based agent applications, made simple and flexible |
| NeMo-RL | NVIDIA-NeMo/RL | 2,033 | 574 | 2025-03-16 | Apache-2.0 | NVIDIA's scalable toolkit for efficient model reinforcement learning |
Three first-pass observations:
- The top two are close in stars but five years apart in age. TRL is the oldest of the five (created 2020-03-27), more than three years ahead of OpenRLHF (2023-07-30) and nearly five years ahead of AReaL and NeMo-RL. We did not sample the upstream verl creation date in this snapshot, but 23,650 stars puts its community mass in the same league as TRL's. An ecosystem veteran that started in 2020 and a fast riser that caught up from behind have very different growth logics — that distinction matters in section three.
- Zero license differentiation. All five are Apache-2.0 (subject to each repository's LICENSE file as the final authority), so licensing drops out of the decision entirely. The weight shifts to engineering shape and ecosystem fit. The only footnote: mimoagent, a companion repo under Xiaomi's fork umbrella, uses MIT — equally permissive.
- All five are actively maintained. Most recent pushes within the snapshot window: TRL, AReaL, and NeMo-RL on 2026-09-27, Xiaomi's fork on 2026-09-26, and OpenRLHF on 2026-09-17. Nobody is dormant, so the "is it alive?" filter eliminates no one.
2. Positioning and Architecture: Who Synchronizes, Who Goes Async, and Who Backs Whom
The real differences among these five are not "can it do RL" but three axes: synchronous versus asynchronous design, single-machine versus multi-machine priorities, and which ecosystem each project is anchored to. Descriptions below stay at official-README levels of caution — features not documented in our sources are not claimed here.
verl: the largest community, now with production-grade validation from Xiaomi
verl's official positioning is HybridFlow — "a flexible and efficient RL post-training framework." It is a ByteDance-lineage open-source project now maintained under the verl-project organization. Its hold on the top star spot matters because when "training agents on verifiable rewards like executable tests and rule checks" turned from a research demo into an industrial requirement, the community's default answer was verl.
Xiaomi's fork adds a layer of production validation. On 2026-09-21 Xiaomi created XiaomiMiMo/verl (465 stars and 49 forks in our snapshot, Apache-2.0, last pushed 2026-09-26). Its README describes it as the Agentic RL training code for MiMo, built on verl 0.9.0.dev and adding reproduction code for five RL environment suites:
| Domain | Task family | Verifier | Launch script |
|---|---|---|---|
| Code | Software engineering | Executable tests | scripts/code/train.sh |
| Cyber | Vulnerability reproduction | Rule checks | scripts/arvo/arvo.sh |
| General | Knowledge work | Rubric review | scripts/general/general.sh |
| Visual | Web development | Visual scoring | scripts/design/webdev.sh |
| Music | Symbolic music composition | Rule checks | scripts/design/music.sh |
Two companion repos were created the same day: uni-agent (14 stars, Apache-2.0, a framework for training long-horizon agents) and mimoagent (30 stars, MIT, a 100-line agent that solves GitHub issues). With the MiMo-V2.6-RL-oss training dataset openly downloadable on Hugging Face, the stack forms a complete training bundle. Read alongside Xiaomi's stated figures — roughly six days of live RL, about 750,000 trajectories, 1M training context, single-step updates of 2.7-3.7B tokens — the fork looks like genuine production infrastructure, not a launch-day publicity move. One caveat: GitHub compare shows the fork at ahead_by 0 and behind_by 16, so divergence details cannot be verified; this article sticks to README-verifiable facts. Training-recipe details belong to Xiaomi's technical report; for integration-level guidance see our MiMo-V2.6 integration SOP.
TRL: the Hugging Face official default with the broadest ecosystem
19,398 stars, 3,021 forks, created March 2020 — the most senior of the five. The official positioning is one line: "train transformer language models with reinforcement learning." That is the broadest framing here; it does not presume you are training agents. RLHF and preference-alignment work — the "make the model behave better" category — all falls within its scope. With Hugging Face's official stewardship and native fit with transformers, TRL is almost always the first name individuals or small teams meet when trying LoRA-scale RLHF. The trade-off: once your goal escalates from alignment to large-scale verifiable-reward Agentic RL, whether its abstractions still fit your cluster size is something to verify with your own workload — your proof-of-concept will answer that.
OpenRLHF: the Ray-based multi-machine workhorse
10,045 stars, created 2023-07-30. The official description compresses the key facts into one sentence: "an easy-to-use, scalable and high-performance Agentic RL framework based on Ray (PPO & DAPO & REINFORCE++)." Unpacked: building on Ray delegates distributed orchestration to general-purpose infrastructure rather than a homegrown scheduler — a direct plus for teams already running Ray in production. Listing PPO, DAPO, and REINFORCE++ in the repository description advertises the supported mainstream RL algorithms. For teams running large-scale RL post-training across machines without wanting to marry a single vendor's stack, OpenRLHF is the middle path: more training-focused than TRL, lighter on conceptual overhead than verl's HybridFlow abstraction.
AReaL: the Ant-backed flag bearer for asynchronous agent RL
5,796 stars, created 2025-02-24, official motto "the RL Bridge for LLM-based Agent Applications, made simple and flexible," with an Ant Group lineage. Our source material tags it as emphasizing asynchronous design, and that tag deserves unpacking. As agent trajectories grow longer — Xiaomi's stated training context is 1M tokens — and environment interactions grow more expensive, a synchronous generate-then-train loop leaves compute idle waiting for the environment. Asynchronous designs decouple generation from training so they run in parallel, which is widely treated as the dividing line for agent RL frameworks. Xiaomi's fully asynchronous architecture in MiMo-V2.6 training corroborates the same trend. If environment interaction dominates your training workload — long-horizon tasks, real tool calls, multi-turn exchanges — AReaL's positioning lands exactly on that point.
NeMo-RL: NVIDIA's toolkit, the enterprise-stack ticket
2,033 stars, created 2025-03-16 — the youngest and smallest by stars of the five, but backed by the NVIDIA-NeMo organization and described officially as a scalable toolkit for efficient model reinforcement learning. Its value logic differs: teams pick it not for community rankings but because their infrastructure already lives inside the NVIDIA family, where the fit among training, inference, and operations tooling is hard to replicate. A low star count in this context means a narrow-and-deep user base, not weakness. If your stack is already committed to NeMo, shortlist it; if not, it naturally ranks lower.
3. Scenario-Based Verdicts
Compressing the analysis into four directly actionable answers:
- Individuals or small teams trying LoRA-scale RLHF: pick TRL. Officially maintained by Hugging Face, the broadest ecosystem, seamless with transformers workflows, and the lowest trial-and-error cost of the five. Get "where do rewards come from and how does data flow" working in TRL first, then decide whether you need something heavier.
- Large-scale verifiable-reward Agentic RL: put verl and OpenRLHF into your PoC. Both grew up for exactly this scenario. verl brings the HybridFlow abstraction, the largest community, and now Xiaomi's production validation plus five environment suites worth copying; OpenRLHF brings the Ray foundation and an algorithm lineup of PPO, DAPO, and REINFORCE++ written on its front door. Run both on your own tasks and your own cluster, then decide on measured throughput.
- Long-horizon asynchronous agent training: AReaL. Asynchrony is its reason for being. The longer the trajectories and the costlier the environment interactions, the larger the payoff. Xiaomi's publicly stated fully asynchronous architecture lends further weight to the direction.
- NVIDIA all-in enterprise stacks: NeMo-RL. If your infrastructure is already inside the NeMo ecosystem, keeping the training framework in the same ecosystem minimizes operational and version-compatibility costs.
Two cross-cutting reminders:
- TRL and verl/OpenRLHF are not rivals so much as stages. A common team pattern is two-phase: use TRL for day-to-day alignment iteration and hand large-scale verifiable-reward Agentic RL to verl or OpenRLHF, switching by project stage.
- The value of Xiaomi's five environment suites is not benchmark numbers but copyable homework. How verifiers plug in (executable tests, rule checks, rubric review, visual scoring), how task families are organized, what launch scripts look like — it is all there to study. Even if you ultimately do not use verl, reading that directory structure will save you a week of design from scratch.
4. Licenses and Risks
All five are Apache-2.0 (per the GitHub API license field), which means commercial use, modification, and redistribution face no extra licensing hurdles — unusual tidiness for this space. Three red lines still need drawing:
- Treat each repository's LICENSE file as the final authority. The GitHub API license field is auto-detected and may diverge from the actual LICENSE file; before commercial deployment, read the original text, especially patent and trademark clauses. The same applies to the MIT license on Xiaomi's companion repo mimoagent.
- Stars are a snapshot, not a trend. The 2026-09-27 snapshot indicates accumulated popularity, not maintenance quality or responsiveness. With creation dates up to five years apart, cross-time star comparisons are inherently skewed; judging momentum requires at least two time points.
- All performance claims are self-reported. Speedup ratios and throughput figures in each README were measured under the project's own harness and task selection. They are not comparable and should not drive your selection. The only reliable method is running your own benchmark on your own cluster.
Conclusion
The biggest takeaway from this comparison is not who won but that five distinct tracks have matured: TRL owns alignment on-ramps, verl and OpenRLHF own large-scale verifiable-reward training, AReaL owns asynchronous long-horizon agents, and NeMo-RL closes out the enterprise stack. Xiaomi forking verl adds a verifiable data point to the question of whether open-source frameworks can carry production-grade Agentic RL — the 750,000-trajectory claim is on the record, and the fork's five environment suites are open for inspection. Map the four scenario verdicts onto your situation, then test with your own workloads. Stars will move; the division of labor between these tracks will not.
FAQ
Q1: Can these five frameworks be benchmarked against each other directly?
A1: No. Each project uses its own evaluation harness and task setup, and all performance numbers are self-reported in their respective READMEs, so they are not comparable across projects. This article deliberately avoids benchmark comparisons and evaluates positioning, ecosystem, and engineering shape instead. For selection decisions, run your own tasks on your target cluster.
Q2: Are star counts a valid basis for choosing a framework?
A2: Only as a reference. The star counts here are a GitHub API snapshot from 2026-09-27, and the five projects' creation dates differ by up to five years, which inherently favors earlier movers. A steadier approach is to look at recent push activity, community responsiveness, and fit with your specific scenario.
Q3: Does Xiaomi forking verl mean verl is the only right answer?
A3: No. The fork is one company's choice for training its own model, and the README itself describes it as the Agentic RL training code for MiMo. It demonstrates that verl can carry production-grade training, but it does not diminish the value of OpenRLHF, AReaL, or the others in their respective scenarios.
Q4: I just want ordinary RLHF alignment — do I need these heavy frameworks?
A4: Usually not. LoRA-scale preference alignment starts and ends with TRL for most teams. Only when your workload involves large-scale trajectory collection, executable verifiers, or multi-machine clusters do verl, OpenRLHF, or AReaL become relevant. For the general fine-tuning route, see our LLM fine-tuning SOP.
Q5: What legal risks should I watch for in commercial use?
A5: All five are Apache-2.0 under the GitHub API's classification, keeping the commercial barrier low, but each repository's LICENSE file remains the final authority. Before commercial deployment, verify the license text directly — particularly patent grant and trademark clauses — and consult legal counsel when needed.