Evaluation results

What is being measured

Sedum generates boilerplate from a provenance record. A record declares intent, constraints, and the files a change is authorized to touch. A generator package declares a team's conventions: which code-injection actions exist, what arguments they take, what they render, and where the result belongs. Sedum hands both to a language model, which replies with a list of action invocations — and then executes that list deterministically.

So the model never writes code. It chooses from a closed vocabulary somebody else declared. These runs measure how well that choosing works, and where it breaks down when it doesn't.

Five rungs, and why more than one is needed

Each rung sits further down the pipeline than the last. The value is in where two of them disagree, not in any one number. A run can select exactly the right actions, bind every argument correctly, and still produce a service that does not boot — and that is a finding about the generator package, not a contradiction to be smoothed over. Each rung below says what it cannot see as plainly as what it measures.

1. 1

   Selection — which actions were chosen?

   The model replies with invocations. Did it reach for the actions a complete answer needs? A reply the pipeline rejects outright is counted separately, as invalid.

   Cannot see: whether the arguments were right. A perfectly selected action with a broken argument scores the same as a correct one.

2. 2

   Binding — what were the arguments set to?

   Scored per argument rather than per invocation, because naming which argument is wrong is the entire point. Four samples once scored a perfect six-of-six on selection while every one of them bound a column default wrongly.

   Cannot see: whether the rendered file is even syntactically valid. t.string :title:string, null: false is a perfect count and broken Ruby.

   Binding is scored by the harness at run time against the case's expectations. That score is not stored in the results file, so this page does not show it — see what this page cannot show.

3. 3

   Form — is the output complete and well-formed?

   Three cheap checks needing no database and no boot: anchor fill (how much of the work the created files declared was accounted for), parses (does the rendered output survive the language's own parser), and idempotent (does applying the same answer twice change anything).

   Cannot see: whether the code does what it is supposed to. params.permit(title, completed) parses cleanly and raises the first time it runs. Anchor fill also cannot tell a model that missed an anchor from a package with an anchor nothing can fill.

4. 4

   Behaviour — does the application build, boot and answer?

   Each valid sample's own choices are applied to a freshly scaffolded application, which is then built, booted, and asserted against over HTTP. Off by default: one behaviour sample costs a scaffold, a dependency install, a database and a boot.

   Cannot see: whether any of it beats not using Sedum at all.

5. 5

   Without Sedum — and what exactly is being removed?

   The same model is asked for the same application with the tooling taken away, and it is taken away in two steps rather than one, because the record and the catalog are not the same thing. Without both rungs a good number above is unfalsifiable — there is nothing it is good compared to.

   baseline keeps the whole record — intent, constraints, and the exact paths it authorizes — and drops the catalog. That is not a model without tooling; it is a model handed a precise specification, and it scores accordingly. intent keeps one sentence and drops everything else, so what it measures is what a sentence alone produces.

   Cannot see: selection, binding, anchor fill or idempotency — there is no vocabulary to count and no anchor to fill. Those cells are blank on this page, never zero. The intent arm is also told no file paths, so it cannot be held to them either.

How to read a rate here

Glossary

valid · invalid · failed

Three outcomes a sample can have, and conflating any two of them hides something. Valid: the model answered and the pipeline accepted the answer. Invalid: it answered and the answer was rejected by a validation rule — a real measurement of the model, reported as its own rate, not an error. Failed: the call never reached the model at all. Failed samples are excluded from every denominator, because an unreachable endpoint is not a model that chose badly; the count is always shown beside the rate that excluded it.

arm — a ladder of what the model was given

Three rungs, and the difference between them is the whole experiment.

• sedum — the provenance record and the generator package's action catalog. The model picks from a declared vocabulary; it does not write code.
• baseline — the record and no catalog. It gets the intent, the constraints, and the exact paths it may write, and produces the files itself.
• intent — the record's intent and nothing else. No constraints, no file list. It decides for itself which files the change even needs.

Both lower rungs get one call and no retry, because re-prompting a build error is a repair loop and this is deliberately not a repair tool. All three share the behaviour question — the application is booted and asserted against, whoever wrote the code — and that is the only rung on which all three are comparable.

tightness — defined vs described

Two generator packages over the same record, differing only in how much the action catalog explains itself. defined declares its arguments and their types; described adds a sentence of prose to each. The axis exists to answer whether a description moves the selection rate.

rejection rules

Why an invalid sample was rejected. The vocabulary is fixed so that a slug written today still resolves a year from now:

• response_shape — the reply was not a decodable invocation list.
• unknown_action — an action no package exposes.
• ambiguous_action — an action more than one package claims.
• missing_kwarg — a required argument the model never mentioned.
• empty_kwarg — a required argument mentioned and left empty. Kept apart from the above on purpose: a value nobody wrote and one deliberately emptied are different mistakes.
• unknown_kwarg — an argument the action does not declare.
• kwarg_type — an argument bound to the wrong type.
• missing_derived_kwarg — an argument the chosen template renders but the invocation never bound, whatever the schema calls optional.
• variant — a discriminator bound to a value with no template behind it.
• unauthorized_path — the invocation would write to a path the record does not authorize.

anchor · anchor fill

A file template plants anchors — named places where injected code belongs. The created files therefore state what work they expect, and anchor fill is the fraction of that work a selection accounted for. Its virtue is that it is derived, not declared: it comes from the package and the record rather than from anyone's hand-written expectation, so it cannot be fitted. Its limit is that an anchor filled by the wrong action still counts as filled, and a package carrying an anchor nothing can target has a ceiling below 100% that is a property of the package, not of the answer. The denominator is planted anchors, not samples.

parses

The rendered output handed to the language's own parser — ruby -c, and so on. Syntax only, never correctness. Code that parses can still raise the first time it runs. The denominator is files checked, not samples.

idempotent

Whether applying the same answer a second time leaves the files unchanged. Sedum replaces regions it owns in place on every rerun, so anything that shifts on a second application is a defect. The denominator is files, not samples.

behaviour: working · disagreed · broke

Kept apart rather than reduced to one rate, because a service that never booted and one that booted and answered wrongly are different findings — a single number would merge a broken generator package with a merely wrong one. Working: every assertion held. Disagreed: it built, booted, answered, and failed at least one assertion — the named assertions are listed on each run. Broke: a phase died, so the assertions never ran.

commit · clean · fixture digest

Because every fixture is vendored into the repository, the commit pins the generator package, the record and Sedum's own code — everything except the model's sampling. It pins none of that if the tree was dirty. The fixture digest is a hash of the packages, records and behaviour target a run was drawn against; two runs with different digests were not asked the same question, so this page never aggregates across them.

calls · retries

A retry budget lets a rejected answer be re-prompted. Raising it improves the headline validity number and — measurably — does not improve the software, which is why first-call validity is reported separately wherever a budget was spent, and why nearly every run here is drawn at zero retries.

what this page cannot show

Selection and binding scores. The harness grades an answer against each case's declared expectations at run time and prints that grade to the terminal, but it stores the raw invocations rather than the grade — deliberately, so that a run can be re-scored later against a sharper question. This page therefore describes what was selected (how often each action was chosen, what it was bound to) and does not grade it. Re-deriving a grade here would be a second implementation of the harness's scoring, and two implementations of one judgement eventually disagree. Run go run ./evals/cmd/eval <case> for the graded report.

one number nothing here draws

The authorized-paths rate on the intent arm. That arm is given no list of files, so its answer is accepted whatever paths it chooses — nothing can be missing and nothing can be unexpected. The rate would therefore be 1.00 on every intent run forever, by construction rather than by performance. A zero that reads as a score is the mistake the blank cells above exist to avoid; a perfect score that cannot be anything else is the same mistake with the sign flipped, and it is worse, because a blank invites a question and a perfect score ends one. Neither this page nor the terminal report prints it; both show the reason instead.