Skip to content

Record what a number was measured WITH, and compute the statistic it is judged BY - #224

Merged
joslat merged 5 commits into
mainfrom
feat/model-identity-comparability
Sep 19, 2026
Merged

joslat merged 5 commits into
mainfrom
feat/model-identity-comparability

Conversation

@joslat

@joslat joslat commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Two model-free additions to the measurement instrument. No runs, no containers, no spend — both work on artifacts that already exist.

1. Gate the band on the model

Deleting the GPT-5.5 deployment made a latent hole load-bearing.

The scoreboard decided two runs were the same measurement from four fields: corpus sha · judge PROMPT fingerprint · question-id fingerprint · store size. All four can match perfectly across two runs made by different models — a deployment repointed under a fixed name is invisible to every one of them. And the typedmemeval verb wrote no model identity at all; I checked a real artifact.

So the model was an unstated invariant of every number this project has published. Deploy a different one, run a vertical, and its artifact is indistinguishable from the 0.36-lineage runs. The band would absorb it — the contaminated-member failure the store-size test exists to catch, invisible to every gate that exists.

The capture already existed: ProviderBuildId reads the provider's system_fingerprint, and its own remarks call it "the only thing that can tell a reader two runs were never comparable." It fed the other verb's manifest and nothing else. It is safe to record — a backend build, not an endpoint or account, which is why the deployment name deliberately never is.

Unknown is not agreement. Two runs reporting no build are not comparable; they are two runs where the question cannot be answered. Treating unknown as a match would let the gate certify exactly the pairing it exists to catch. Consequence, stated plainly: runs from before this field existed no longer band with each other. They were never known comparable.

That rule exposed a real bug: with unknown-vs-unknown false, a run reporting no build excluded itself from its own band, giving n=0 — a count no run can have. The head is now always its own member.

2. Compute the census — and it corrects three published numbers

The statistic the identity preregs decide on — "1 exact, 14 undercounts, 0 overcounts, 15 of 43 gold recovered" — was derived by hand and lives in correspondence and a code comment, nowhere in the tool. Every report already retains GoldAnswer and AgentResponse per question, so --census <report|directory> recomputes it over runs already paid for.

Confirms the BEFORE arm's structural claim mechanically for the first time: exact 1, undercounts 14, overcounts 0 — one-directional loss with a clean over-merge guard, which is E-1's entire argument.

Corrects three figures, all mine:

published actual
BEFORE gold recovery 15 of 43 (35%) 17 of 45 (38%)
entlink census "FLAT (14→14)" 14→12 undercounts + 3 refusals, recovery 38%→24%
noderank gold recovery 38%→47% (a rise) 38%→36% (a fall — sign inverted)

Conclusions survive and harden: entity linking was worse than reported, not flat.

And one nobody reported: refusals rise monotonically across the three arms, 0 → 3 → 4. Each identity intervention made the system likelier to decline to count — a stronger version of the displacement finding, sitting in the artifacts the whole time.

An asserted zero is a count, not a refusal: "No deliveries at the annexe are recorded" answers with zero and undercounts by the whole gold, while "I don't have any retrieved record" declines. Reading both as silence is what made the first cut disagree with the hand figures.

Verification

Release build 0 warnings; unit 5257, harness 886. Every new test red-probed. Also fixed en route: a literal backspace byte had replaced a \b in one regex, so a refusal pattern could never match — the branch was swept, nothing else carries it.

🤖 Generated with Claude Code

Deleting the deployment made a latent hole load-bearing, so it is closed before any run happens on
a new model rather than after.

WHAT WAS WRONG. The scoreboard decides two runs are the same measurement from four fields: corpus
sha, judge PROMPT fingerprint, question-id fingerprint, and store size. All four can match
perfectly across two runs made by different models — a deployment repointed under a fixed name is
invisible to every one of them. And the typedmemeval verb wrote no model identity at all: I checked
a real artifact, and there is no model, deployment or build field anywhere in it.

So the model was an unstated invariant of every number this project has published — the C-D
scoreboard, the 54/58/64 temporal band, every E-1 arm. The concrete risk is not hypothetical now:
deploy a different model, run one vertical, and its artifact is indistinguishable from the
0.36-lineage runs. The band would absorb it. That is the contaminated-member failure the store-size
test was added to catch, invisible to every gate that exists.

THE CAPTURE ALREADY EXISTED AND NOTHING CONSUMED IT. `ProviderBuildId` reads the provider's
`system_fingerprint`, and its own remarks call it "the only thing that can tell a reader two runs
were never comparable". It reaches the OTHER verb's preparation manifest. The verb that produced
the scoreboard wrote it nowhere. Eleven dark levers, and this is the same shape one level up: a
value recorded, documented as decisive, and not read by the decision it exists to inform.

It is safe to write. `system_fingerprint` identifies a backend BUILD, not an endpoint, account or
resource — which is why it can live in an artifact meant to be shared while the deployment name
deliberately never does.

UNKNOWN IS NOT AGREEMENT, and that is the whole design. Two runs reporting no build are not
comparable; they are two runs about which the question cannot be answered. Treating unknown as a
match would let the gate certify exactly the pairing it exists to catch — which is how the first
temporal band admitted its contaminated member: every field it checked agreed, and the field that
disagreed was not being checked. A consequence worth stating plainly: runs made before this field
existed no longer band with each other. They were never KNOWN to be comparable, and saying so is
the scoreboard's job.

That rule then exposed a real bug in the banding predicate: with unknown-vs-unknown false, a run
reporting no build excluded ITSELF from its own band and the row read n=0 — a count no run can
have. The head is now always its own member, which is what every other test in that predicate
silently assumed.

Two tests, both red-probed: treating unknown as agreement fails one, removing the gate fails both.
Release build 0 warnings. Unit 5257, harness 871.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 19, 2026 19:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Provider metadata can be aggregated across runs and roles, allowing incomplete or mismatched artifacts to band incorrectly.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 High severity · 1 Medium severity · 1 Low severity

Open (5)
What changed in this PR

This PR adds provider backend-build identity to TypedMemEval provenance and scoreboard banding.

Changes:

  • Captures and persists observed provider builds.
  • Requires matching builds when forming bands and preserves head-run membership.
  • Adds tests for differing and unknown builds.
File Description
TypedMemEvalScoreboard.cs Reads provider-build metadata for band comparison.
TypedMemEvalProgram.cs Collects provider builds and writes provenance.
ScoreboardTests.cs Tests build mismatch and unknown-build behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +182 to +183
() => ProviderBuildsOf(
answerChatClient, judgeChatClient, extractionChatClient))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A baseline is taken when each vertical starts and only that vertical's delta is recorded, so an all invocation no longer attributes earlier verticals' builds to later ones. Red-probed: defeating the baseline fails the scoping test.

Comment on lines +446 to +448
/// Answer, judge and extraction together, because all three shape a score and any of them
/// changing makes two runs incomparable. A run that saw more than one build is already suspect
/// and says so by carrying both.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Recorded by role now — answer, judge, extraction kept apart — and the gate compares role by role, treating a role present in one run and absent in the other as a different configuration rather than a detail. Your A/B-vs-B/A example is the case the test uses.

Comment on lines +462 to +468
private static IReadOnlyList<string> ProviderBuildsOf(params LongMemEvalChatCallMeter?[] meters) =>
meters
.Where(meter => meter is not null)
.SelectMany(meter => meter!.Snapshot().ProviderBuilds.Keys)
.Distinct(StringComparer.Ordinal)
.OrderBy(build => build, StringComparer.Ordinal)
.ToArray();

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, and this was the sharpest of the five: the field was built by breaking the rule the gate enforces. CallsWithoutProviderBuild was captured and dropped, so a partially-unaccounted run recorded a clean identity. Any metered call without a build now makes that vertical UNKNOWN.

Comment on lines +273 to +280
IReadOnlyList<string> providerBuilds =
sidecar.TryGetProperty("providerBuilds", out var builds)
&& builds.ValueKind == JsonValueKind.Array
? [.. builds.EnumerateArray()
.Where(item => item.ValueKind == JsonValueKind.String)
.Select(item => item.GetString()!)
.OrderBy(build => build, StringComparer.Ordinal)]
: [];

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by whole-field rejection rather than element filtering. Skipping the bad element manufactures a clean identity out of a damaged artifact. The run stays placeable and keeps its score — only its comparability is unaccounted for — and simply never bands.

Comment on lines 546 to +547
run = runIndex,
providerBuilds = providerBuildBlock,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, and you are right that this was the real gap: producer and consumer were each tested and the seam between them by neither, which is this codebase's oldest failure shape. There is now a provider-free producer test covering per-role capture, baseline scoping, and unknown propagation, each red-probed.

THE DECISION STATISTIC WAS DERIVED BY A PERSON READING A REPORT. "1 exact, 14 undercounts, 0
overcounts, 15 of 43 gold recovered" appears in two preregs, in correspondence, and in a code
comment -- and nowhere in the tool. A statistic nobody can re-derive cannot be checked when it is
doubted, cannot be red-probed, and its Goodhart guard is a promise rather than a check.

Every TypedMemEval report already retains GoldAnswer and AgentResponse per question, so this needed
no run and no model. `--census <report|directory>` recomputes it over artifacts already paid for.

WHAT IT CONFIRMS. The BEFORE arm is exact 1, undercounts 14, overcounts 0 -- the structural claim
E-1 rests on, one-directional loss with a clean over-merge guard, verified mechanically for the
first time.

WHAT IT CORRECTS, all three mine:

  1. Gold recovery was "15 of 43 (35%)". Summing the fifteen gold answers gives 45, and recovery is
     17 -- 38%. Two hand-derived versions of this quantity were already in circulation, a prereg
     saying 35% and correspondence saying 38%; they disagreed and nobody noticed.
  2. "CENSUS FLAT (14 -> 14)" for entity linking is wrong. Undercounts fell 14 -> 12 while THREE
     answers became refusals. The conclusion stands and hardens -- gold recovery fell 38% -> 24%,
     so the arm was worse than reported, not flat.
  3. Node-distance re-ranking's gold recovery was published as 38% -> 47%, a rise. It is 38% -> 36%,
     a fall. The sign was inverted.

AND ONE NOBODY REPORTED: refusals rise monotonically across the three arms, 0 -> 3 -> 4. Each
identity intervention made the system likelier to decline to count. That is a stronger version of
the displacement finding than the one published, and it was sitting in artifacts the whole time.

AN ASSERTED ZERO IS A COUNT, NOT A REFUSAL. "No deliveries at the annexe are recorded" answers with
zero and undercounts by the whole gold; "I don't have any retrieved record" declines. Both carry no
digit, and the first cut of this census read both as silence -- which is precisely what made it
disagree with the hand figures until the distinction was drawn. The line is what the sentence is
about: a claim about the record answers, a claim about the assistant's own reach does not.

Found and fixed while writing this: a literal backspace byte had replaced a `\b` in one pattern, so
a refusal test could never match. The branch was swept for the same damage; nothing else carries it.

15 tests. Release build 0 warnings. Unit 5257, harness 886. No model calls, no container, no spend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@joslat joslat changed the title Gate the band on the model, which was an invariant nothing recorded Record what a number was measured WITH, and compute the statistic it is judged BY Sep 19, 2026
joslat and others added 3 commits September 19, 2026 23:33
…hmetic

Caught by running the new verb over every artifact rather than only the one it was written for.

Unrestricted, it reported `arithmetic-sum` with eight "overcounts" and `arithmetic-delta` with
seven. Their golds are MAGNITUDES -- 3,012 and 3,297 -- so answering high is an arithmetic error,
not two referents merged into one. Firing the over-merge guard there is a category error, and a
guard that fires on nearly every artifact teaches its reader to skip the line. This codebase
already learned that once: a warning printed where a zero is the correct and expected state is how
a gate stops being read at all.

The over-merge signature is specific. Identity capture that merges distinct referents makes a COUNT
OF THINGS rise past the number of things there are, which is only meaningful where the answer counts
items. Membership is decided by the shape's own name -- a named suffix, not a heuristic, in the same
spirit as the supersession list that is deliberately a named list of one.

After the restriction the guard fires on two shapes across roughly thirty artifacts instead of
nearly all of them, which is a signal rather than noise.

TWO THINGS THE SWEEP SURFACED, both free:

  - `arithmetic-count` reaches 14/14 exact and 100% gold recovery on the ORACLE arm, against 30-56%
    on every real arm. The oracle reads projected gold directly, so that shape is fully answerable
    and its gap is engine, not corpus.
  - Unparseable answers are pervasive on `arithmetic-count`, between 3 and 14 of 14. The refusal
    signal the conjunction arms showed is not peculiar to identity work.

23 tests. Release build 0 warnings. Unit 5257, harness 894. No model calls, no spend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five findings, all real, and two of them cut at the design rather than the code.

THE FIELD CONTRADICTED THE GATE IT FEEDS. `CallsWithoutProviderBuild` was captured by the meter and
dropped on the floor. If nine responses carried `fp_a` and a tenth carried nothing, the artifact
recorded `fp_a` and the gate called the run fully comparable -- while the gate's own remarks, three
commits old, say unknown is not agreement. The rule was enforced on the field and broken while
building it. A call anywhere in a vertical that reports no build now makes that vertical's identity
UNKNOWN, which is the honest answer to "which build produced this": part of it is unaccounted for.

A FLATTENED SET CANNOT TELL TWO SYSTEMS APART. Answer and judge on A with extraction on B, and the
reverse, both serialise as [A, B]. Those are different configurations and would have banded.
Recorded by ROLE now, and the gate compares role by role -- a role present in one run and absent
from the other is a different configuration, not a detail.

THE METERS OUTLIVE THE VERTICAL. They are created once and reused across an `all` invocation, so
reading their totals at the end of the third vertical reported builds the first two saw. After any
backend rollover a clean run would look mixed, and clean runs would stop banding -- the exact
opposite of the point. A baseline is taken when a vertical starts and only its own delta is
recorded.

A MALFORMED FIELD WAS SILENTLY REPAIRED INTO A VALID IDENTITY. `["fp_a", 123]` read as `["fp_a"]`
and could band. Whole-field rejection now: a reader that skips the bad element manufactures a clean
identity out of a damaged artifact, which is worse than admitting it cannot tell. The run keeps its
score and simply never bands -- only its comparability is unaccounted for, not its result.

AND THE PRODUCER WAS TESTED BY NOBODY. The scoreboard tests hand-wrote the field; the meter tests
stopped at Snapshot(). A regression between them would have left both suites green while real
artifacts omitted or mis-scoped it. That is this codebase's oldest failure shape -- two halves
tested apart and the seam tested by neither -- and it is now covered: per-role capture, baseline
scoping, and unknown propagation, each red-probed. Defeating the baseline fails the scoping test;
defeating the missing-call check fails the unknown test.

Release build 0 warnings. Unit 5257, harness 900. No model calls, no spend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… bug

Found by review, no run. The plan files this as "make write-side supersession fire on the eval
profile"; the reason it never fired is one rung further out than that.

THE CHAIN, end to end. `CanSupersede` requires a CANONICAL single-valued predicate -- six of them:
belongs to, costs, expires, lives in, weighs, works at. Without the relation vocabulary the
extractor invents a predicate per sentence, measured elsewhere in this codebase at 700 facts under
421 distinct predicates. `usePredicateVocabulary` exists on the profile, DEFAULTS FALSE, and the
typedmemeval verb could not set it. So every TypedMemEval run this project has ever made extracted
free-form predicates, none could be recognised as replacing anything, `:SUPERSEDED_BY` was
unwritable BY CONSTRUCTION, and the vertical whose entire subject is supersession has been
measuring its own off-state since the day it was added.

The bitemporal store's own numbers say it plainly: 6,839 facts, 0 superseded edges, 0 invalidated,
and its commonest predicates are speech acts -- `came up with` x1482, `said` x1263, `was consistent
between` x490. Not one of the six appears.

THE CORPUS IS NOT THE PROBLEM. It asserts a genuinely single-valued state and then corrects it:
"Alice Renwick is at Calderwick, as of February", later "that earlier note was wrong -- it was
Lowick from February", with questions asking what the record shows NOW and what it showed AS OF 19
January. That is valid time against transaction time, exactly what the feature is for.

AND I AM NOT PATCHING THE LEXICON, which is the tempting fix. Extraction rendered that state as
`is at` x198, which is not a declared surface form of `lives in`. Adding it would make every "is at"
single-valued -- and a new one would then SUPERSEDE the old, dropping a true fact from live recall
with nothing marking that it had been closed. That is the data-shaped defect the cardinality class
warns about in its own remarks, it would be tuning a shipped lexicon to one corpus, and it would
damage every consumer of this library to move one benchmark row. Steering extraction is the
designed mechanism; widening the lexicon is not.

So the lever is wired: `--predicate-vocabulary`, off by default, its own arm token `vocab` because
it changes what ingestion records. A bitemporal enablement arm is now nameable as
`supersede-vocab` and dry-runs clean over all 60 questions.

WHAT THIS DOES NOT CLAIM. Wiring the flag does not prove supersession will fire. The vocabulary
tells the extractor to PREFER those relations; whether it then writes `lives in` for this corpus is
an empirical question, and there is no budget to answer it. What is now true is that the question
CAN be asked, which it could not be before -- and that the ⛔ off-state is explained rather than
merely observed.

Release build 0 warnings. Unit 5257, harness 905. No model calls, no spend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@joslat
joslat merged commit b2d7d49 into main Sep 19, 2026
7 checks passed
@joslat
joslat deleted the feat/model-identity-comparability branch September 19, 2026 22:06
joslat added a commit that referenced this pull request Sep 21, 2026
…ry number

Sixteen benchmark programs each read AZURE_OPENAI_ENDPOINT/_API_KEY/_DEPLOYMENT
inline and built their own AzureOpenAIClient. That is sixteen places for a
provider decision to be made slightly differently, and it is exactly why the
extraction override was honoured in five of them and ignored in the rest.
HarnessClients resolves once and hands out clients by ROLE.

THE ROLES ARE NOT INTERCHANGEABLE, and the mechanical pass got that wrong before
this commit fixed it. The answer client is the subject under test and keeps the
operator's timeout so a hung model fails fast. The judge is NOT the subject:
timing it out discards a measurement already paid for, so it gets a generous
timeout and honours AI_JUDGE_*. Seven programs were building their judge from
the answer model, which is how a self-graded result comes to look independent.

RUN IDENTITY. PR #224 stamped the model and the backend build and not the HOST.
The same model id served by two providers is not the same measurement --
different quantisation, different serving stack, different sampling defaults --
and without the host in the identity two such artifacts are indistinguishable.
Every place that recorded a deployment name now records model@provider, and
embeddings model@provider/dims, because the width is part of the store.

The dimension probe now CROSS-CHECKS rather than being replaced by the resolved
value. The probe is ground truth -- it asks the model that will write the
vectors -- but it cannot notice the store was built at a different width; the
resolved value is what configures the store but comes from a shipped table. When
they disagree one of them is wrong about an index that cannot report its own
corruption, so it refuses instead of picking a winner. Passing the resolved
value is also what keeps it from being a parameter nothing feeds.

Sample header comments said AZURE_OPENAI_* was "required". It is not, and a
comment that says so is worse than none.

Post-change grep: every AZURE_OPENAI survivor is prose, a docs row, the
package's own azure branch, the parity alias, or a test asserting the
backward-compatible path. No code outside AgentMemory.Inference reads one.

Release 0 warnings; 5314 + 905 + 54 + 42 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants