Repository navigation
Record what a number was measured WITH, and compute the statistic it is judged BY - #224
Conversation
Deleting the deployment made a latent hole load-bearing, so it is closed before any run happens on a new model rather than after. WHAT WAS WRONG. The scoreboard decides two runs are the same measurement from four fields: corpus sha, judge PROMPT fingerprint, question-id fingerprint, and store size. All four can match perfectly across two runs made by different models — a deployment repointed under a fixed name is invisible to every one of them. And the typedmemeval verb wrote no model identity at all: I checked a real artifact, and there is no model, deployment or build field anywhere in it. So the model was an unstated invariant of every number this project has published — the C-D scoreboard, the 54/58/64 temporal band, every E-1 arm. The concrete risk is not hypothetical now: deploy a different model, run one vertical, and its artifact is indistinguishable from the 0.36-lineage runs. The band would absorb it. That is the contaminated-member failure the store-size test was added to catch, invisible to every gate that exists. THE CAPTURE ALREADY EXISTED AND NOTHING CONSUMED IT. `ProviderBuildId` reads the provider's `system_fingerprint`, and its own remarks call it "the only thing that can tell a reader two runs were never comparable". It reaches the OTHER verb's preparation manifest. The verb that produced the scoreboard wrote it nowhere. Eleven dark levers, and this is the same shape one level up: a value recorded, documented as decisive, and not read by the decision it exists to inform. It is safe to write. `system_fingerprint` identifies a backend BUILD, not an endpoint, account or resource — which is why it can live in an artifact meant to be shared while the deployment name deliberately never does. UNKNOWN IS NOT AGREEMENT, and that is the whole design. Two runs reporting no build are not comparable; they are two runs about which the question cannot be answered. Treating unknown as a match would let the gate certify exactly the pairing it exists to catch — which is how the first temporal band admitted its contaminated member: every field it checked agreed, and the field that disagreed was not being checked. A consequence worth stating plainly: runs made before this field existed no longer band with each other. They were never KNOWN to be comparable, and saying so is the scoreboard's job. That rule then exposed a real bug in the banding predicate: with unknown-vs-unknown false, a run reporting no build excluded ITSELF from its own band and the row read n=0 — a count no run can have. The head is now always its own member, which is what every other test in that predicate silently assumed. Two tests, both red-probed: treating unknown as agreement fails one, removing the gate fails both. Release build 0 warnings. Unit 5257, harness 871. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Provider metadata can be aggregated across runs and roles, allowing incomplete or mismatched artifacts to band incorrectly.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
Open (5)
What changed in this PR
This PR adds provider backend-build identity to TypedMemEval provenance and scoreboard banding.
Changes:
- Captures and persists observed provider builds.
- Requires matching builds when forming bands and preserves head-run membership.
- Adds tests for differing and unknown builds.
| File | Description |
|---|---|
TypedMemEvalScoreboard.cs |
Reads provider-build metadata for band comparison. |
TypedMemEvalProgram.cs |
Collects provider builds and writes provenance. |
ScoreboardTests.cs |
Tests build mismatch and unknown-build behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| () => ProviderBuildsOf( | ||
| answerChatClient, judgeChatClient, extractionChatClient)) |
There was a problem hiding this comment.
Fixed. A baseline is taken when each vertical starts and only that vertical's delta is recorded, so an all invocation no longer attributes earlier verticals' builds to later ones. Red-probed: defeating the baseline fails the scoping test.
| /// Answer, judge and extraction together, because all three shape a score and any of them | ||
| /// changing makes two runs incomparable. A run that saw more than one build is already suspect | ||
| /// and says so by carrying both. |
There was a problem hiding this comment.
Fixed. Recorded by role now — answer, judge, extraction kept apart — and the gate compares role by role, treating a role present in one run and absent in the other as a different configuration rather than a detail. Your A/B-vs-B/A example is the case the test uses.
| private static IReadOnlyList<string> ProviderBuildsOf(params LongMemEvalChatCallMeter?[] meters) => | ||
| meters | ||
| .Where(meter => meter is not null) | ||
| .SelectMany(meter => meter!.Snapshot().ProviderBuilds.Keys) | ||
| .Distinct(StringComparer.Ordinal) | ||
| .OrderBy(build => build, StringComparer.Ordinal) | ||
| .ToArray(); |
There was a problem hiding this comment.
Fixed, and this was the sharpest of the five: the field was built by breaking the rule the gate enforces. CallsWithoutProviderBuild was captured and dropped, so a partially-unaccounted run recorded a clean identity. Any metered call without a build now makes that vertical UNKNOWN.
| IReadOnlyList<string> providerBuilds = | ||
| sidecar.TryGetProperty("providerBuilds", out var builds) | ||
| && builds.ValueKind == JsonValueKind.Array | ||
| ? [.. builds.EnumerateArray() | ||
| .Where(item => item.ValueKind == JsonValueKind.String) | ||
| .Select(item => item.GetString()!) | ||
| .OrderBy(build => build, StringComparer.Ordinal)] | ||
| : []; |
There was a problem hiding this comment.
Fixed by whole-field rejection rather than element filtering. Skipping the bad element manufactures a clean identity out of a damaged artifact. The run stays placeable and keeps its score — only its comparability is unaccounted for — and simply never bands.
| run = runIndex, | ||
| providerBuilds = providerBuildBlock, |
There was a problem hiding this comment.
Fixed, and you are right that this was the real gap: producer and consumer were each tested and the seam between them by neither, which is this codebase's oldest failure shape. There is now a provider-free producer test covering per-role capture, baseline scoping, and unknown propagation, each red-probed.
THE DECISION STATISTIC WAS DERIVED BY A PERSON READING A REPORT. "1 exact, 14 undercounts, 0
overcounts, 15 of 43 gold recovered" appears in two preregs, in correspondence, and in a code
comment -- and nowhere in the tool. A statistic nobody can re-derive cannot be checked when it is
doubted, cannot be red-probed, and its Goodhart guard is a promise rather than a check.
Every TypedMemEval report already retains GoldAnswer and AgentResponse per question, so this needed
no run and no model. `--census <report|directory>` recomputes it over artifacts already paid for.
WHAT IT CONFIRMS. The BEFORE arm is exact 1, undercounts 14, overcounts 0 -- the structural claim
E-1 rests on, one-directional loss with a clean over-merge guard, verified mechanically for the
first time.
WHAT IT CORRECTS, all three mine:
1. Gold recovery was "15 of 43 (35%)". Summing the fifteen gold answers gives 45, and recovery is
17 -- 38%. Two hand-derived versions of this quantity were already in circulation, a prereg
saying 35% and correspondence saying 38%; they disagreed and nobody noticed.
2. "CENSUS FLAT (14 -> 14)" for entity linking is wrong. Undercounts fell 14 -> 12 while THREE
answers became refusals. The conclusion stands and hardens -- gold recovery fell 38% -> 24%,
so the arm was worse than reported, not flat.
3. Node-distance re-ranking's gold recovery was published as 38% -> 47%, a rise. It is 38% -> 36%,
a fall. The sign was inverted.
AND ONE NOBODY REPORTED: refusals rise monotonically across the three arms, 0 -> 3 -> 4. Each
identity intervention made the system likelier to decline to count. That is a stronger version of
the displacement finding than the one published, and it was sitting in artifacts the whole time.
AN ASSERTED ZERO IS A COUNT, NOT A REFUSAL. "No deliveries at the annexe are recorded" answers with
zero and undercounts by the whole gold; "I don't have any retrieved record" declines. Both carry no
digit, and the first cut of this census read both as silence -- which is precisely what made it
disagree with the hand figures until the distinction was drawn. The line is what the sentence is
about: a claim about the record answers, a claim about the assistant's own reach does not.
Found and fixed while writing this: a literal backspace byte had replaced a `\b` in one pattern, so
a refusal test could never match. The branch was swept for the same damage; nothing else carries it.
15 tests. Release build 0 warnings. Unit 5257, harness 886. No model calls, no container, no spend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hmetic
Caught by running the new verb over every artifact rather than only the one it was written for.
Unrestricted, it reported `arithmetic-sum` with eight "overcounts" and `arithmetic-delta` with
seven. Their golds are MAGNITUDES -- 3,012 and 3,297 -- so answering high is an arithmetic error,
not two referents merged into one. Firing the over-merge guard there is a category error, and a
guard that fires on nearly every artifact teaches its reader to skip the line. This codebase
already learned that once: a warning printed where a zero is the correct and expected state is how
a gate stops being read at all.
The over-merge signature is specific. Identity capture that merges distinct referents makes a COUNT
OF THINGS rise past the number of things there are, which is only meaningful where the answer counts
items. Membership is decided by the shape's own name -- a named suffix, not a heuristic, in the same
spirit as the supersession list that is deliberately a named list of one.
After the restriction the guard fires on two shapes across roughly thirty artifacts instead of
nearly all of them, which is a signal rather than noise.
TWO THINGS THE SWEEP SURFACED, both free:
- `arithmetic-count` reaches 14/14 exact and 100% gold recovery on the ORACLE arm, against 30-56%
on every real arm. The oracle reads projected gold directly, so that shape is fully answerable
and its gap is engine, not corpus.
- Unparseable answers are pervasive on `arithmetic-count`, between 3 and 14 of 14. The refusal
signal the conjunction arms showed is not peculiar to identity work.
23 tests. Release build 0 warnings. Unit 5257, harness 894. No model calls, no spend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five findings, all real, and two of them cut at the design rather than the code. THE FIELD CONTRADICTED THE GATE IT FEEDS. `CallsWithoutProviderBuild` was captured by the meter and dropped on the floor. If nine responses carried `fp_a` and a tenth carried nothing, the artifact recorded `fp_a` and the gate called the run fully comparable -- while the gate's own remarks, three commits old, say unknown is not agreement. The rule was enforced on the field and broken while building it. A call anywhere in a vertical that reports no build now makes that vertical's identity UNKNOWN, which is the honest answer to "which build produced this": part of it is unaccounted for. A FLATTENED SET CANNOT TELL TWO SYSTEMS APART. Answer and judge on A with extraction on B, and the reverse, both serialise as [A, B]. Those are different configurations and would have banded. Recorded by ROLE now, and the gate compares role by role -- a role present in one run and absent from the other is a different configuration, not a detail. THE METERS OUTLIVE THE VERTICAL. They are created once and reused across an `all` invocation, so reading their totals at the end of the third vertical reported builds the first two saw. After any backend rollover a clean run would look mixed, and clean runs would stop banding -- the exact opposite of the point. A baseline is taken when a vertical starts and only its own delta is recorded. A MALFORMED FIELD WAS SILENTLY REPAIRED INTO A VALID IDENTITY. `["fp_a", 123]` read as `["fp_a"]` and could band. Whole-field rejection now: a reader that skips the bad element manufactures a clean identity out of a damaged artifact, which is worse than admitting it cannot tell. The run keeps its score and simply never bands -- only its comparability is unaccounted for, not its result. AND THE PRODUCER WAS TESTED BY NOBODY. The scoreboard tests hand-wrote the field; the meter tests stopped at Snapshot(). A regression between them would have left both suites green while real artifacts omitted or mis-scoped it. That is this codebase's oldest failure shape -- two halves tested apart and the seam tested by neither -- and it is now covered: per-role capture, baseline scoping, and unknown propagation, each red-probed. Defeating the baseline fails the scoping test; defeating the missing-call check fails the unknown test. Release build 0 warnings. Unit 5257, harness 900. No model calls, no spend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… bug Found by review, no run. The plan files this as "make write-side supersession fire on the eval profile"; the reason it never fired is one rung further out than that. THE CHAIN, end to end. `CanSupersede` requires a CANONICAL single-valued predicate -- six of them: belongs to, costs, expires, lives in, weighs, works at. Without the relation vocabulary the extractor invents a predicate per sentence, measured elsewhere in this codebase at 700 facts under 421 distinct predicates. `usePredicateVocabulary` exists on the profile, DEFAULTS FALSE, and the typedmemeval verb could not set it. So every TypedMemEval run this project has ever made extracted free-form predicates, none could be recognised as replacing anything, `:SUPERSEDED_BY` was unwritable BY CONSTRUCTION, and the vertical whose entire subject is supersession has been measuring its own off-state since the day it was added. The bitemporal store's own numbers say it plainly: 6,839 facts, 0 superseded edges, 0 invalidated, and its commonest predicates are speech acts -- `came up with` x1482, `said` x1263, `was consistent between` x490. Not one of the six appears. THE CORPUS IS NOT THE PROBLEM. It asserts a genuinely single-valued state and then corrects it: "Alice Renwick is at Calderwick, as of February", later "that earlier note was wrong -- it was Lowick from February", with questions asking what the record shows NOW and what it showed AS OF 19 January. That is valid time against transaction time, exactly what the feature is for. AND I AM NOT PATCHING THE LEXICON, which is the tempting fix. Extraction rendered that state as `is at` x198, which is not a declared surface form of `lives in`. Adding it would make every "is at" single-valued -- and a new one would then SUPERSEDE the old, dropping a true fact from live recall with nothing marking that it had been closed. That is the data-shaped defect the cardinality class warns about in its own remarks, it would be tuning a shipped lexicon to one corpus, and it would damage every consumer of this library to move one benchmark row. Steering extraction is the designed mechanism; widening the lexicon is not. So the lever is wired: `--predicate-vocabulary`, off by default, its own arm token `vocab` because it changes what ingestion records. A bitemporal enablement arm is now nameable as `supersede-vocab` and dry-runs clean over all 60 questions. WHAT THIS DOES NOT CLAIM. Wiring the flag does not prove supersession will fire. The vocabulary tells the extractor to PREFER those relations; whether it then writes `lives in` for this corpus is an empirical question, and there is no budget to answer it. What is now true is that the question CAN be asked, which it could not be before -- and that the ⛔ off-state is explained rather than merely observed. Release build 0 warnings. Unit 5257, harness 905. No model calls, no spend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ry number Sixteen benchmark programs each read AZURE_OPENAI_ENDPOINT/_API_KEY/_DEPLOYMENT inline and built their own AzureOpenAIClient. That is sixteen places for a provider decision to be made slightly differently, and it is exactly why the extraction override was honoured in five of them and ignored in the rest. HarnessClients resolves once and hands out clients by ROLE. THE ROLES ARE NOT INTERCHANGEABLE, and the mechanical pass got that wrong before this commit fixed it. The answer client is the subject under test and keeps the operator's timeout so a hung model fails fast. The judge is NOT the subject: timing it out discards a measurement already paid for, so it gets a generous timeout and honours AI_JUDGE_*. Seven programs were building their judge from the answer model, which is how a self-graded result comes to look independent. RUN IDENTITY. PR #224 stamped the model and the backend build and not the HOST. The same model id served by two providers is not the same measurement -- different quantisation, different serving stack, different sampling defaults -- and without the host in the identity two such artifacts are indistinguishable. Every place that recorded a deployment name now records model@provider, and embeddings model@provider/dims, because the width is part of the store. The dimension probe now CROSS-CHECKS rather than being replaced by the resolved value. The probe is ground truth -- it asks the model that will write the vectors -- but it cannot notice the store was built at a different width; the resolved value is what configures the store but comes from a shipped table. When they disagree one of them is wrong about an index that cannot report its own corruption, so it refuses instead of picking a winner. Passing the resolved value is also what keeps it from being a parameter nothing feeds. Sample header comments said AZURE_OPENAI_* was "required". It is not, and a comment that says so is worse than none. Post-change grep: every AZURE_OPENAI survivor is prose, a docs row, the package's own azure branch, the parity alias, or a test asserting the backward-compatible path. No code outside AgentMemory.Inference reads one. Release 0 warnings; 5314 + 905 + 54 + 42 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju



Two model-free additions to the measurement instrument. No runs, no containers, no spend — both work on artifacts that already exist.
1. Gate the band on the model
Deleting the GPT-5.5 deployment made a latent hole load-bearing.
The scoreboard decided two runs were the same measurement from four fields: corpus sha · judge PROMPT fingerprint · question-id fingerprint · store size. All four can match perfectly across two runs made by different models — a deployment repointed under a fixed name is invisible to every one of them. And the typedmemeval verb wrote no model identity at all; I checked a real artifact.
So the model was an unstated invariant of every number this project has published. Deploy a different one, run a vertical, and its artifact is indistinguishable from the 0.36-lineage runs. The band would absorb it — the contaminated-member failure the store-size test exists to catch, invisible to every gate that exists.
The capture already existed:
ProviderBuildIdreads the provider'ssystem_fingerprint, and its own remarks call it "the only thing that can tell a reader two runs were never comparable." It fed the other verb's manifest and nothing else. It is safe to record — a backend build, not an endpoint or account, which is why the deployment name deliberately never is.Unknown is not agreement. Two runs reporting no build are not comparable; they are two runs where the question cannot be answered. Treating unknown as a match would let the gate certify exactly the pairing it exists to catch. Consequence, stated plainly: runs from before this field existed no longer band with each other. They were never known comparable.
That rule exposed a real bug: with unknown-vs-unknown false, a run reporting no build excluded itself from its own band, giving
n=0— a count no run can have. The head is now always its own member.2. Compute the census — and it corrects three published numbers
The statistic the identity preregs decide on — "1 exact, 14 undercounts, 0 overcounts, 15 of 43 gold recovered" — was derived by hand and lives in correspondence and a code comment, nowhere in the tool. Every report already retains
GoldAnswerandAgentResponseper question, so--census <report|directory>recomputes it over runs already paid for.Confirms the BEFORE arm's structural claim mechanically for the first time: exact 1, undercounts 14, overcounts 0 — one-directional loss with a clean over-merge guard, which is E-1's entire argument.
Corrects three figures, all mine:
entlinkcensusnoderankgold recoveryConclusions survive and harden: entity linking was worse than reported, not flat.
And one nobody reported: refusals rise monotonically across the three arms, 0 → 3 → 4. Each identity intervention made the system likelier to decline to count — a stronger version of the displacement finding, sitting in the artifacts the whole time.
An asserted zero is a count, not a refusal: "No deliveries at the annexe are recorded" answers with zero and undercounts by the whole gold, while "I don't have any retrieved record" declines. Reading both as silence is what made the first cut disagree with the hand figures.
Verification
Release build 0 warnings; unit 5257, harness 886. Every new test red-probed. Also fixed en route: a literal backspace byte had replaced a
\bin one regex, so a refusal pattern could never match — the branch was swept, nothing else carries it.🤖 Generated with Claude Code