On September 1, 2026, Anthropic released Claude Fable 5.1 and Mythos 5.1, publishing Terminal-Bench 4.0 scores of 55.8% for Fable 5.1, 60.9% for Mythos 5.1, and 37.3% for GPT-5.6 Sol. That same day, harbor-framework/harbor and harbor-framework/terminal-bench both received code pushes. The coincidence is two sides of one thing: vendors publish scores on leaderboards, while the tasks and framework producing those scores are themselves open source. This article covers the second half, how to use that toolchain to produce numbers of your own, turning a claimed score from something you can only quote into something you can reproduce. For the release, see Claude Fable 5.1 and Mythos 5.1: What Changed. For the bill, see Agentic Cost Accounting Under a Cache-First Architecture.
1. Why Vendor Scores Can't Be Used Directly
Those official numbers have three things in common: run by the vendor, published by the vendor, measured on one benchmark using the vendor's own agent configuration. Any one of them is enough to make a number a signal rather than a conclusion.
Self-reported methodology. Third parties can rarely reproduce a vendor score in full. What harness was used, what the system prompt said, which tools were allowed, whether failures were retried: these details almost never make it into an announcement. Without them you cannot tell whether a result is the ceiling of the model's capability or the output of one tuned configuration.
Safety layers in production. Vendors run production traffic behind permission boundaries, command interception, and output filtering, and usually leave unsaid whether the benchmark ran with those layers on. The gap is not a rounding error: an operation blocked by a filter counts as a failure on the scoreboard, while in a real workflow it may be the right behavior.
An opaque harness. The same model on the same task can score very differently under different agent frameworks, because tool sets, step limits, context management, and parallel tool calling all differ. This is also why cross-vendor scores are not directly comparable.
The point is not that scores are useless. It is that a score only carries decision weight once you have re-run it under your own terms.
2. What Harbor and Terminal-Bench Actually Are
Based on repository descriptions and homepages: harbor-framework/harbor calls itself a "Framework for evaluating and improving agents," while harbor-framework/terminal-bench is "Measuring and evolving with the frontier of agent work," homepage tbench.ai. Both sit under the same organization and the descriptions place them upstream and downstream: Harbor is the framework and toolchain for evaluating and improving agents; Terminal-Bench is the task and benchmark suite that runs on it. Neither project states this in one official sentence, so treat it as an inference.
The analogy that helps: Harbor is the track, the stopwatch, and the record keeping, standardizing what it means to run one evaluation. Terminal-Bench is the exam paper and marking scheme, defining what gets tested and what counts as a pass. Swap models, frameworks, or prompts and you still run the same exam under the same timing rules, the only condition under which scores are comparable.
That division dictates what you may change. Comparing models means holding the exam fixed and changing only the subject; checking whether your agent improved means the reverse. Change both at once and the number explains nothing.
3. A Map of Four Repos and One Rename
Get the repositories straight before you start. One rename matters: the original laude-institute/terminal-bench now redirects (301) to harbor-framework/terminal-bench-1, and the organization moved from laude-institute to harbor-framework. Any old link in your docs or scripts now lands on terminal-bench-1, which is not the actively developed terminal-bench.
| Repository | Stars | Forks | Language | License | Created | Last push |
|---|---|---|---|---|---|---|
| harbor-framework/harbor | 4872 | 1705 | Python | Apache-2.0 | 2025-08-04 | 2026-09-01 |
| harbor-framework/terminal-bench | 594 | 434 | Python | Apache-2.0 | 2026-01-25 | 2026-09-01 |
| harbor-framework/terminal-bench-1 | 2559 | 567 | Python | Apache-2.0 | 2025-01-17 | 2026-07-11 |
| harbor-framework/terminal-bench-2 | 393 | 112 | Shell | Apache-2.0 | 2025-09-25 | 2026-04-30 |
Three traps hide in those rows. First, the repo with the most stars is not the one still being updated. terminal-bench-1 has 2559 stars but last saw a push on 2026-07-11; terminal-bench-2 has 393 stars and stopped on 2026-04-30; meanwhile terminal-bench, with just 594 stars, was pushed to on 2026-09-01. Stars are accumulated history, push time is current activity. Second, terminal-bench, terminal-bench-1, and terminal-bench-2 are three distinct repositories, not versions of one thing, so never compare a score from one against another. Third, all four are Apache-2.0 per the GitHub API, not MIT.
Two actions follow. Replace any hardcoded laude-institute/terminal-bench clone URL with the post-migration repository, and whenever you cite a score record which repo, which version, and what date, because three months later you will not know where that 55.8% came from.
4. Recalibration: Old Scores Are Not Comparable
Terminal-Bench 4.0 is one benchmark Anthropic used to evaluate Fable 5.1, and it did something that matters for how you read the numbers: it recalibrated the time, CPU, and memory quotas for every task, removed 8 tasks, and fixed 19. Anthropic states explicitly that 4.0 scores are not comparable to earlier versions.
What does a quota change do? Pass rates on agentic tasks are extremely sensitive to resource limits. Tighten the time budget and the model may not have failed to solve the task, it may simply have run out of clock. Tighten memory and heavier tasks never start. That changes the conditions for completing the exam, not the capability of the model. So when you see two scores for one model on "Terminal-Bench" from different dates, ask which version each came from.
A second pair is worth reading alongside: on Terminal-Bench-Science 0.1, a research-oriented set, Fable 5.1 scored 52.6% and Fable 5 scored 24.7%. That is a brand-new benchmark and the gap is large, but it is 0.1 and will change too. Fine as evidence that capability is rising, not as evidence that your workload improves by the same margin.
The takeaway: write the benchmark version into your evaluation config and archive it with the results. The version number is the unit of the score, and two numbers without units cannot be subtracted.
5. A Four-Step Path to Your Own Evaluation
These steps depend on no command I cannot verify. For installation and invocation, rely on the repository README and official documentation.
Step one: write your own tasks before running the full suite. A public benchmark's distribution rarely matches your production distribution. Take ten to thirty tasks you have actually hit in production, build them into a private set, and get the pipeline working first; then bring in the public benchmark for cross-calibration. The private set answers "does this work for my workloads," the public set "where do we sit relative to the industry."
Step two: put the harness configuration under version control. Everything that moves the score must be versioned: model identifier, system prompt, tool list, step limit, retry policy, sandbox quotas, and parallel tool calling. Every config change must trace to a specific run, or you will have no trail when scores drift.
Step three: repeat runs, report distributions, not points. Agentic tasks carry real randomness and a single pass rate may just be luck. Run each configuration several times, record the range rather than one number, and log time and cost per run. Fix the repeat count once and stop changing it.
Step four: treat cost and latency as first-class metrics. An evaluation that measures only pass rate rewards finishing the task at any price. A complete table looks like this:
| Dimension | How to measure | Why it is mandatory |
|---|---|---|
| Pass rate | Repeat runs, report a range not a point | A single result cannot distinguish real gains from luck |
| Latency per run | End-to-end wall clock and in-task step count | Decides whether it fits an interactive path |
| Cost per run | Aggregate on token and tool-call accounting | A higher pass rate may simply have been bought |
| Failure modes | Classify as timeout, tool misuse, context exhaustion, environment error | Different modes need completely different fixes |
| Degraded path | Run once with fallback triggered and once without | Production may not be on the optimal path |
That last row is easiest to skip, yet a real share of production traffic may run on the fallback path while your evaluation only exercised the happy one. The fallback architecture itself is a separate piece.
6. Six Traps and How to Avoid Them
-
Treating a benchmark score as a service commitment. It is one measurement on a fixed task set under fixed quotas. It says nothing about your distribution and is not an SLA.
-
Ignoring harness differences. Swap the framework or prompt at the same time as the model and the score change cannot be attributed. One variable at a time.
-
Running it once. See step three. A single run cannot separate a real gain from luck.
-
Not pinning versions. Task suites change, 4.0 removed 8 tasks and fixed 19. Frameworks and model identifiers change too. Pin before you ship, and treat an upgrade as a change to evaluate.
-
Excluding breaking API changes. Model-side API changes alter agent behavior directly, and problems like losing the reasoning chain on fallback never surface unless your evaluation builds a degraded path. See Claude Fable 5.1 API Migration SOP.
-
Measuring capability but not cost. A pass rate that rises while cost per run doubles can be a net loss. See Agentic Cost Accounting Under a Cache-First Architecture.
7. Who This Is For, and Where It Falls Short
It fits three groups: teams evaluating models without vendor announcements, teams running agents in production that need weekly regression testing, and anyone who has to explain internally why a model looks worse in their environment than on the leaderboard. For all three the value is the same: it moves the conversation from "whose score is higher" back to "what is our own number."
The limits deserve plain statement. First, building your own evaluation is not cheap: writing a private task set, pinning a harness, and running repeats cost people and compute, so small teams can start with ten to thirty tasks. Second, my description of what these projects do comes from their own descriptions and homepages; for concrete APIs, install steps, and configuration keys, defer to the official README. This article gives no operational command I could not verify. Third, and most fundamental: a self-built evaluation solves the comparability problem, not the capability problem. It tells you how a model performs on your tasks, but it cannot make a weak model strong.
A vendor's benchmark score is a starting point, not an ending point. Whether an agent works in your terminal is decided by your tasks, your harness, and your bill. Wire up this toolchain, run ten to thirty real tasks, and turn the score into a number you reproduced yourself. Selection debates get quieter after that.
References
- GitHub REST API, measured 2026-09-02: harbor-framework/harbor (4872 stars / 1705 forks / Python / Apache-2.0 / created 2025-08-04 / last push 2026-09-01 / https://harborframework.com/); harbor-framework/terminal-bench (594 stars / 434 forks / Python / Apache-2.0 / created 2026-01-25 / last push 2026-09-01 / https://tbench.ai); harbor-framework/terminal-bench-1 (2559 stars / 567 forks / Python / Apache-2.0 / created 2025-01-17 / last push 2026-07-11 / https://www.tbench.ai); harbor-framework/terminal-bench-2 (393 stars / 112 forks / Shell / Apache-2.0 / created 2025-09-25 / last push 2026-04-30, no description).
- Repository rename: the original laude-institute/terminal-bench now redirects (301) to harbor-framework/terminal-bench-1, and the organization moved from laude-institute to harbor-framework (GitHub REST API, measured 2026-09-02).
- Anthropic official publication, released 2026-09-01: Terminal-Bench 4.0 scores of 55.8% for Claude Fable 5.1, 60.9% for Mythos 5.1, and 37.3% for GPT-5.6 Sol. Version 4.0 recalibrated per-task time, CPU, and memory quotas, removed 8 tasks, and fixed 19, so scores are not comparable to earlier versions. Announcement URL unconfirmed.
- Anthropic official publication: Terminal-Bench-Science 0.1 scores of 52.6% for Claude Fable 5.1 and 24.7% for Claude Fable 5. Announcement URL unconfirmed.
- The positioning of Harbor relative to Terminal-Bench is inferred from repository descriptions and homepage information; neither project states the relationship in one official sentence, flagged as inference in the body text.
- Beyond the items above, the remainder is engineering recommendation and inference, flagged in place, not official source material.
FAQ
Q1: Are Harbor and Terminal-Bench the same project? A1: No. They are two repositories under the same harbor-framework organization. Based on repository descriptions and homepage information, Harbor is a framework for evaluating and improving agents while Terminal-Bench is the task and benchmark suite: one provides the toolchain for running evaluations, the other defines what is tested and what counts as a pass. Neither project defines the relationship in one official sentence, so this is an inference.
Q2: Why do instructions built around laude-institute/terminal-bench no longer line up? A2: Because that repository now redirects (301) to harbor-framework/terminal-bench-1, and the organization moved as well. Old URLs still resolve, but they land on a repo last pushed to on 2026-07-11, not the actively developed harbor-framework/terminal-bench pushed to on 2026-09-01. Replace hardcoded URLs in your scripts once, and record the repository name and retrieval date whenever you cite a score.
Q3: Can I reproduce the vendor's 55.8% and 60.9% in my own environment? A3: Probably not, and you should not try. Those numbers were produced under a harness, task quotas, and safety configuration the vendor selected, details that are typically not published in full. The goal is not to reproduce the vendor's number but to get one on your own task set that is repeatable, attributable, and trackable over time.
Q4: Did Terminal-Bench 4.0 get harder or easier than earlier versions? A4: Neither framing works. Version 4.0 recalibrated the time, CPU, and memory quotas for every task, removed 8 tasks, and fixed 19, and Anthropic states that scores are not comparable to earlier versions. A quota change alters the conditions for completing the exam, not the absolute capability of the model. Before accepting any cross-version comparison, confirm both sides used the same version.
Q5: What is the smallest viable evaluation for a small team? A5: Ten to thirty tasks you have genuinely hit in production, one pinned harness configuration committed to version control, several repeat runs per configuration reported as a range, and four metrics logged every time: pass rate, latency, cost, and failure mode classification. Get those solid before expanding the task set or bringing in a public benchmark.