Skip to content

Give QueryEvaluator the Content fallback, and pin both query providers to one corpus (#3511) - #3534

Merged
meshweaver-cloud[bot] merged 2 commits into
mainfrom
fix/3511-query-provider-selector-parity
Sep 7, 2026
Merged

meshweaver-cloud[bot] merged 2 commits into
mainfrom
fix/3511-query-provider-selector-parity

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Closes #3511.

The defect, as the issue finally scoped it

The query language has ONE grammar and TWO executors, in two repositories:

executor runs for
A QueryEvaluator (core) in-memory / FileSystem / static-node hosts — and, on every backend, live-query relevance (PostgreSqlPartitionedMeshQuery.IsRelevant) and merged-result ordering (MeshQuery.ClipMergedInitial)
B PostgreSqlSqlGenerator.MapSelector (MeshWeaver.Plugins) SQL against the content JSONB on every portal

Nothing compared them. MapSelector has always resolved known column → content.X walk → n.content->>'X'. QueryEvaluator had the first two and not the third: an unknown selector reflected against MeshNode and returned null, so the predicate matched nothing.

Measured on a live Postgres mesh, 2026-09-07 — the numbers that re-scoped the issue:

query Postgres QueryEvaluator
nodeType:NodeType compilationStatus:Error 5 0, always
nodeType:NodeType compilationStatus:Ok 195, disjoint from the 5 0, always
nodeType:NodeType content.compilationStatus:Error the same 5 the same 5

🚨 The negated form was worse than blind: CompareEqual(null, "Ok") is false, so NotEqual inverted to true — -compilationStatus:Ok matched every node, and -status:Decommissioned matched every deployment. A filter written to exclude returned what it was written to exclude.

The fix

GetPropertyValue resolves the first hop as the object's own property, else the same name out of Content — the rule SQL always had. compilationStatus is named nowhere; it is reachable because the rule reaches it.

Three decisions, each stated in the code:

  1. The node's own field WINS; content is only the fallback. This is MapSelector's order and the only order that changes nothing already working. Content-first would silently re-point every live query whose selector names both — and name, description, category, icon, order, state, version are common content field names. This way round, the only selectors whose resolution moves are the ones that resolved to null — i.e. matched nothing.
  2. Keys on the property being ABSENT, never on its value being null. SQL answers the n.description column for description: and never falls through. GetDirectPropertyValue is split into a presence-reporting TryGetDirectPropertyValue.
  3. First hop only — SQL's default is a single root-level n.content->>'X'.

The test that compares the two providers

test/MeshWeaver.Fixture/SelectorResolutionCorpus.cs — pure data, in a project both repos reference. Per selector: the side each provider reads it from, and the exact column expression SQL must emit. Core asserts QueryEvaluator resolves every row on its recorded side; the Plugins twin asserts MapSelector emits every row's SqlExpression. Same shape as the existing QueryRouteClassifier. Divergences are named, not merely absent.

The #3515 control (a healthy type answers Ok, not Error) is kept, and a second control added on the negation arm.

Falsification — both arms, with numbers

result
fallback removed 7 failed / 30 passed of 37
fallback restored 37 / 37
full MeshWeaver.Hosting.Test 441 / 441
MeshWeaver.Graph.Test 1383 / 1383
MeshWeaver.Documentation.Test (all ratchet/static/async guards) 304 / 304

dotnet build -c Release -warnaserror clean, one project per invocation, on MeshWeaver.Hosting, MeshWeaver.Fixture, MeshWeaver.Hosting.Test, MeshWeaver.Graph.Test, MeshWeaver.Documentation.Test.

🚨 A third defect, found by checking the blast radius (second commit)

sort: resolves through the same GetPropertyValue, and MeshQuery.ClipMergedInitial runs OrderResults to order merged results on every backend, Postgres included. Before the fallback a bare content selector resolved to null for every row, so every sort key was null and nothing could be mis-compared. The fallback widens that to the content's real values — and JSON is not type-uniform. Reproduced deterministically before fixing:

InvalidOperationException: Failed to compare two elements in the array
 ---> ArgumentException: Object must be of type String

So a widened selector would have turned a merely differently-ordered query into a failed one, on the merge path. (The same latent hole already swallowed int against long; nothing reached it because nothing sorted on a mixed key.)

OrderResults now orders through a total comparer. Uniform keys keep their existing order exactly — same type + IComparable is dispatched first and untouched — so only the pairs the default comparer refused are newly decided: mixed numerics numerically, anything else by its invariant string form (ordinal, never the ambient culture). Postgres has no equivalent hazard: n.content->>'X' is always TEXT.

🚨 Two more defects found on the way

1. #3515's assertions could not fail. GetPropertyValue(...)?.ToString().Should().Be("Error") short-circuits the whole chain on null — .Should() is never reached and the test passes having asserted nothing. Both stayed GREEN against the un-fixed evaluator. Now stringified through interpolation (null renders ""). That repair is what took the red arm from 5 failures to 7.

2. PropertyMap omits six real columns. createdBy, createdDate, lastModifiedBy, desiredId, syncBehavior, excludeFromContext are all real mesh_nodes columns (AuthorColumns, SyncBehaviorColumn, ExcludeFromContextColumn) that MapSelector does not list, so Postgres reads the content field of the same name instead. Same shape as Plugins#1310. createdBy is in live use (the chat history selector). This is a Plugins fix; recorded in KnownDivergences and pinned so it cannot grow silently.

The AGENTS.md sweep string — and a corrected premise

Changed to search 'nodeType:NodeType content.compilationStatus:Error' — not because the bare form stays broken, but because the sweep runs against whatever image a portal is ALREADY running, where the fix has not shipped.

🚨 The issue's premise that this line sits in a shared-rule block is FALSE. Core's AGENTS.md carries exactly two blocks — rest-not-graphql (L142–143) and conserve-work-products (L145–146). The sweep line is L55, outside both, and no satellite carries the string at all (measured over all six via REST: Education/Reinsurance/SocialMedia/Manufacturing each have one unrelated compilationStatus mention; Plugins and Memex have none). So this changes ONE file and check-shared-rules.py is not engaged.

Work products committed

  • Doc/Architecture/QueryProviderParity (new) — the two executors, what the drift cost, the ordering decision, the corpus, the residual divergence table, and the third resolver (MeshSearchView.razor.cs, which falls back on null VALUE rather than absence and should delegate).
  • Doc/DataMesh/QuerySyntax — the three-step selector-resolution rule as an author sees it.
  • Doc/Architecture/BuildIdentityAdmission — corrected off "the evaluator is pinned rather than changed".
  • What's New entry (Category: Fix) — live results not refreshing, content-field sorts being no-ops, and -status:X returning what it excluded.

Pairs-with: none — no public type or member is removed (GetPropertyValue's signature is unchanged; everything added). The Plugins follow-up (widen PropertyMap, add the corpus-driven MapSelector test, make MeshSearchView's third resolver delegate) needs this merged plus a pin bump first, and is filed as Systemorph/MeshWeaver.Plugins#1439.

🤖 Generated with Claude Code

…s to one corpus (#3511)

The query language has ONE grammar and TWO executors, in two repositories:
QueryEvaluator here (in-memory, FileSystem and static-node hosts -- and, on
EVERY backend, live-query relevance via PostgreSqlPartitionedMeshQuery.IsRelevant
and merged-result ordering via MeshQuery.ClipMergedInitial) and
PostgreSqlSqlGenerator.MapSelector in MeshWeaver.Plugins. Nothing compared them.

MapSelector has always resolved a selector as: known column, else content.X walk,
else n.content->>'X'. QueryEvaluator had the first two and not the third -- an
unknown selector reflected against MeshNode and returned null, so the predicate
matched NOTHING. Measured on a live Postgres mesh 2026-09-07:
compilationStatus:Error returned 5, compilationStatus:Ok returned 195, disjoint;
the same queries matched nothing in the evaluator. The negated form was worse
than blind: CompareEqual(null, "Ok") is false, so -compilationStatus:Ok matched
EVERY node -- a filter written to exclude returned what it excluded.

The fix is the RULE, not a special case; compilationStatus is named nowhere.

Ordering, and why: the object's own property wins and content is only the
fallback. That is MapSelector's order (PropertyMap before the n.content->>
default) and the only order that changes nothing already working -- content-first
would re-point every live query whose selector names both, and name/description/
category/icon/order/state/version are common content field names. This way round
the only selectors whose resolution moves are the ones that resolved to null,
i.e. matched nothing. The fallback keys on the property being ABSENT, never on
its value being null (GetDirectPropertyValue split into a presence-reporting
TryGetDirectPropertyValue), because SQL answers the n.description column for
description: and never falls through. First hop only, for the same parity.

SelectorResolutionCorpus (test/MeshWeaver.Fixture, referenced by both repos) is
the shared statement of the rule: per selector, the side each provider reads it
from and the exact column expression SQL must emit. Core asserts QueryEvaluator
resolves every row on its recorded side; the Plugins twin asserts MapSelector
emits every row's SqlExpression. Divergences are named, not merely absent.

Falsification, both arms: with the fallback removed 7 of 36 fail; restored, 36/36
pass. Full project 440/440, documentation suite 304/304.

Two defects found on the way:
- #3515's assertions used `GetPropertyValue(...)?.ToString().Should().Be(...)`,
  which short-circuits the whole chain on null -- .Should() is never reached and
  the test passes having asserted nothing. Both stayed GREEN against the un-fixed
  evaluator. Now stringified through interpolation (null renders ""), which is
  what took the red arm from 5 failures to 7.
- MapSelector's PropertyMap omits createdBy/createdDate/lastModifiedBy/desiredId/
  syncBehavior/excludeFromContext although all six are real mesh_nodes columns, so
  Postgres reads the content field of the same name. Same shape as Plugins#1310.
  Recorded in KnownDivergences; the fix is a Plugins change.

AGENTS.md's pre-deploy sweep becomes content.compilationStatus:Error -- not
because the bare form stays broken, but because the sweep runs against whatever
image a portal is ALREADY running. The line is NOT in a shared-rule block (only
rest-not-graphql and conserve-work-products are) and no satellite carries it, so
this changes one file.

Doc: Doc/Architecture/QueryProviderParity, the selector-resolution rule in
Doc/DataMesh/QuerySyntax, and BuildIdentityAdmission corrected off "pinned rather
than changed".

Closes #3511

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 7, 2026 05:48
@meshweaver-cloud
meshweaver-cloud Bot enabled auto-merge September 7, 2026 05:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes correctly align selector resolution semantics across providers and are backed by stronger, control-based tests; only minor naming nits remain.

Pull request overview

This PR fixes a long-standing parity gap between the in-memory query executor (QueryEvaluator in core) and the Postgres SQL translator (MapSelector in MeshWeaver.Plugins) by adding a first-hop Content fallback in QueryEvaluator and introducing a shared selector-resolution corpus to keep both providers pinned to the same behavior. It also updates operational guidance and documentation so the recommended sweep/query spellings remain correct across deployed images and backends.

Changes:

  • Add first-hop selector resolution fallback in QueryEvaluator (node property first, then Content of the same name) to match SQL behavior and fix negation behaving “worse than blind” for unknown selectors.
  • Introduce SelectorResolutionCorpus (shared fixture data) and expand/strengthen the existing sweep selector test to validate the new resolution rule and its controls.
  • Update AGENTS guidance and documentation to describe the three-step selector resolution rule and the operationally safe sweep query form.
File summaries
File Description
test/MeshWeaver.Hosting.Test/SweepSelectorReachesTheCompileStatusTest.cs Expands tests to cover the new fallback behavior, ordering decision, negation control, and validates resolver behavior against the shared corpus.
test/MeshWeaver.Fixture/SelectorResolutionCorpus.cs Adds a shared selector-resolution corpus (data-only) to pin both providers’ behavior to one canonical list, including named divergences.
src/MeshWeaver.Hosting/Persistence/Query/QueryEvaluator.cs Implements first-hop Content fallback with presence-aware property resolution (distinguishing absent vs null).
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-07-searching-on-a-content-field-means-one-thing-everywhere.md Adds a user-facing “Fix” entry describing the behavioral impact (live results, sorting, negation filters).
src/MeshWeaver.Documentation/Data/DataMesh/QuerySyntax.md Documents the three-step selector resolution rule and recommends the explicit content.X form for operational stability.
src/MeshWeaver.Documentation/Data/Architecture/QueryProviderParity.md New architecture doc explaining the two-provider model, the drift, the fix, the shared corpus, and remaining divergences.
src/MeshWeaver.Documentation/Data/Architecture/BuildIdentityAdmission.md Updates the build identity admission guidance to reflect the fixed evaluator and the still-recommended dotted sweep form.
AGENTS.md Updates the prescribed pre-prod sweep query to use content.compilationStatus and explains why the prefix remains required operationally.
Review details

Suppressed comments (1)

test/MeshWeaver.Hosting.Test/SweepSelectorReachesTheCompileStatusTest.cs:129

  • Test method name contains an all-caps run ("HOLDING") which reduces readability and isn’t consistent with typical method naming in this test project. Consider renaming to standard PascalCase.
    public void AKnownFieldHOLDINGNullDoesNotFallIntoContent()
  • Files reviewed: 8/8 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Fact]
public void TheBareSelectorDoesNotReachIt_WhichIsWhyAGreenSweepHereIsNotEvidence()
=> new QueryEvaluator().GetPropertyValue(BrokenType(), "compilationStatus")
public void ANodeFieldBEATSASameNamedContentField()
@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

251 tests  ±0   251 ✅ ±0   2m 14s ⏱️ +13s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

492 tests  ±0   492 ✅ ±0   43s ⏱️ -2s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 034 tests  ±0   1 034 ✅ ±0   1m 12s ⏱️ +2s
    2 suites ±0       0 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 600 tests  ±0   1 406 ✅ ±0   2m 10s ⏱️ +2s
    4 suites ±0     194 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±0      5 suites  ±0   2m 14s ⏱️ ±0s
1 473 tests ±0  1 473 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 474 runs  ±0  1 474 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

♻️ This comment has been updated with latest results.

…chable (#3511)

Found while checking the blast radius of the Content fallback, and reproduced
deterministically before fixing.

`sort:` resolves through the same GetPropertyValue as a filter, and
MeshQuery.ClipMergedInitial runs OrderResults to order merged results on EVERY
backend, Postgres included. Before the fallback a bare content selector resolved
to null for every row, so every sort key was null and nothing could be
mis-compared. The fallback widens that to the content's real values -- and JSON
is not type-uniform: the same key is a string on one node and a number on
another. Comparer<object>.Default answers that pair with

  InvalidOperationException: Failed to compare two elements in the array
   ---> ArgumentException: Object must be of type String

so a widened selector would turn a merely differently-ordered query into a FAILED
one, on the merge path every backend runs. The same latent hole already swallowed
int against long; nothing reached it because nothing sorted on a mixed key.

OrderResults now orders through a total comparer. Uniform keys keep their
existing order EXACTLY -- same type + IComparable is dispatched first and
untouched, so strings still sort as strings and ticks still sort chronologically.
Only the pairs the default comparer REFUSED are newly decided: mixed numerics
numerically, anything else by its invariant string form (ordinal, never resolved
from the ambient culture). Postgres has no equivalent hazard because
n.content->>'X' is always TEXT.

SortingOnAContentFieldOfMixedTypes pins it; it was written first and observed to
fail with the exception above.

Falsification re-measured against the final suite: fallback removed 7 of 37 fail,
restored 37/37. MeshWeaver.Hosting.Test 441/441, MeshWeaver.Graph.Test 1383/1383,
MeshWeaver.Documentation.Test 304/304.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 113 tests  +34   1 113 ✅ +34   3m 28s ⏱️ -2s
    3 suites ± 0       0 💤 ± 0 
    3 files   ± 0       0 ❌ ± 0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Test Results

   16 files  ± 0     16 suites  ±0   12m 3s ⏱️ +13s
5 963 tests +34  5 769 ✅ +34  194 💤 ±0  0 ❌ ±0 
5 964 runs  +34  5 770 ✅ +34  194 💤 ±0  0 ❌ ±0 

Results for commit e71ffa3. ± Comparison against base commit 75a062f.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 7, 2026
Merged via the queue into main with commit 459e35f Sep 7, 2026
29 checks passed
@rbuergi
rbuergi deleted the fix/3511-query-provider-selector-parity branch October 10, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants