- Yardsticks in playFour, all external
- Checks per sweep217,060 + 1,451 + 833
- RegeneratedEvery release
- Release gatesFive, automatic
Ask any software company how they know a release is better than the last one, and you will usually get an answer about process — stand-ups, reviews, sprints. Those are useful. They are not evidence.
On Raku++, our from-scratch compiler for the Raku programming language, we ran the other experiment: decide almost nothing by opinion. Pick yardsticks we do not control, measure against them on every release, publish the results including the ones that make us look bad, and let the graphs choose what gets worked on next.
This page is that method, with the actual numbers. Every chart below is drawn from the live dashboards behind raku.online/spec — the same data the project's own engineers look at.
1. Pick a yardstick you do not control
The temptation with an in-house test suite is that you write the tests and you decide when they are good enough. That is a mirror, not a measurement. Raku happens to have a rare asset: its specification is executable — around 1,460 files of runnable tests, maintained by the language's own community, entirely outside our control.
So that became the primary yardstick from day one. Not "does it feel like Raku" but "how many of someone else's assertions pass".
Source: the project's release dashboard. A file counts only if every assertion in it passes — the strictest of the three measures available.
That is a satisfying line, and it is also the least interesting one on this page. A rising graph is easy to draw. What matters is what happens when the graph tells you something you did not want to hear.
2. Understand what your number does when you improve
Here is the same project, same releases, measured as a percentage of the tests the suite declares:
Source: the project's release dashboard. 157,293 of 191,546 at v0.5.0; 196,395 of 217,060 at v1.5.0.
The work between those first two releases was some of the most productive in the project. The percentage went down.
The reason is structural, and it is worth understanding because the same trap exists in most projects. When a test file crashes early, the tests after the crash never get declared. Fix the crash and the file runs further — declaring dozens more tests, most of which now fail. You got better; your denominator got bigger faster.
There are three defensible ways to compute that pass rate. We publish the strictest one, always alongside the whole-file figure, with the methodology written down. The alternative — quietly picking whichever definition flatters this week's release — is how measurement turns into marketing.
3. Know how much your number wobbles
A number without an error bar invites people to over-read it. Two sweeps of an identical code tree gave us 934, 939, and 943 agreements on the same data set. Nothing changed between the runs — the reference engine randomises hash iteration order per process, so a handful of examples flip in both directions.
So the rule on this project is: a movement inside the noise band is not a result. We recorded the band, and treat anything under about ±5 on that measure as flat.
| What we track | Scale | How noisy | What counts as real |
|---|---|---|---|
| Specification tests passing | 217,060 | ±20 tests, ±1 file | A named fix, gated on no regressions |
| Documentation examples matching | 1,451 | ±5, iteration order | A movement outside the band |
| Operator checks matching | 833 | Deterministic | Any change at all |
| Benchmark kernels | 9 | A few % run to run | Best of six, against a recorded baseline |
4. Compare against a reference, example by example
A specification suite tests features in isolation. It does not tell you whether your product behaves the way the documentation tells users it will. So we built a second yardstick: take every runnable example in the language's official documentation — 1,451 of them — run each on our engine and on the mature reference implementation, and compare the output byte for byte.
That produces more than a pass/fail. Each example lands in one of several buckets, and the buckets are the useful part:
Source: the conformance sweep behind raku.online/spec, four regenerations between 25 and 29 July 2026. 1,451 examples per sweep.
Read the red band: that is our defect list, and it fell from 471 to 120 in four regenerations. Every one of those was a real behavioural difference someone had to find, understand, and close.
But look at the pale blue band, the one that grew — from 104 to 123. Those are examples where our engine and the reference implementation agree with each other and both disagree with the documentation. In other words: the documentation is out of date. Measuring against two independent references at once turns "we have a bug" into "we have a bug, they have a bug, or the docs are wrong" — and separating those three is most of the work.
5. Then go finer, one operator at a time
Whole-example comparisons find clusters but not causes. So the same treatment goes one level down: 833 individual expressions across 121 operators, each evaluated on both engines and diffed.
Source: the operator behaviour matrix behind raku.online/spec. “Both reject” means neither engine accepts the expression — agreement of a different kind.
Thirty divergences left, from seventy-two, in a single release cycle. And the reason that collapse was so steep is exactly why fine-grained measurement pays: several of those fixes were one root cause each clearing fifteen rows at once. One parser bug made a term be read as a function call, which broke every operator that could follow it. You do not find that by fixing symptoms — you find it by having a table where fifteen unrelated-looking rows go green together.
6. Watch the number that drifts
Correctness metrics get attention because they are the ones on the roadmap. Performance is different: it degrades quietly, one reasonable-looking change at a time, and nobody notices because no single commit is the culprit.
Here is a benchmark kernel across the project's releases. Lower is better.
Source: the project's benchmark harness, re-measured on every release. Best of six runs on one machine, process startup included.
The first drop is deliberate work — a 2.2× improvement. Then look at the dashed stretch: over six releases the number crept from 769 ms back up to 903 ms. Nobody shipped a performance regression on purpose. Nobody reviewed it either, because nothing was watching.
What fixed it was not heroics but arithmetic: profile, rank the candidates by measured cost, and work the list. Five candidates were ranked from a profile. Three landed. One was implemented, measured, found to be 2.5% slower, and reverted. One was estimated at a 4% ceiling for the riskiest change on the list and deliberately never attempted. The result was 744 ms — faster than the number had ever been.
Then the important part: a recorded baseline is now re-measured on every release and fails the build if it slips. The same drift cannot happen twice, because a number nobody is gating on is a number that will drift.
What we actually do with the numbers
The graphs are the visible part. The habits behind them are what make them worth anything:
- Measure against something external — a standard, a reference implementation, a customer's own acceptance data
- Publish the least flattering defensible figure, and write down how it is computed
- Establish the noise band before reading movements as progress
- Keep a correctness fix even when it lowers the score, and say why the number moved
- Turn every closed defect into a permanent regression test — 149 of them so far
- Gate releases on the metrics automatically, so nothing depends on someone remembering to look
- Record the experiments that failed, so nobody spends a week rediscovering them
None of this is specific to compilers. Substitute "latency at the 99th percentile", "orders that reconcile to the penny", or "documents the new pipeline classifies identically to the old one" and the method is unchanged: an external yardstick, a known noise band, a regenerated graph, and a gate.
Where this fits
Our methodologies page covers how we run projects — the cadence, the communication, the delivery. This page is the other half: how we know the thing we delivered is actually better than what came before.
If your project has a number that matters — a speed budget, a correctness bar, a migration that has to produce identical output — that is work we like, and this is how we would approach it.
The dashboards behind these graphs are public: the specification and its conformance data, and the project's own measurement documentation on GitHub.