Repository navigation
Run every JSON-Schema-Test-Suite runner on the suite's unified layout - #1058
Merged
Merged
Conversation
Contributor
Test Results 48 files 48 suites 16m 51s ⏱️ Results for commit f1020ab. ♻️ This comment has been updated with latest results. |
Contributor
|
The documentation preview for this pull request has been removed. |
…e layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…ignum cases by name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…te layout Each case's compatibility selects the drafts it applies to, its tags whether format is asserted, and its externalSchemas are served to that case alone. Rust leaves out, by name, the six contradictory required leap-second cases and the high-precision bignum cases its serde_json values cannot hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
The C API runner leaves out the same cases as the crate's. The Bowtie IHOP test sends each case's externalSchemas as its registry and leaves out the format-assertion cases, since the harness treats format as an annotation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
… on the unified layout The suite runner expects 8,611 cases (less zero-terminated floats in draft4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…s on the unified layout The suite runner expects 8,611 cases (less zero-terminated floats in draft4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…d layout The suite runner leaves out, by name, the six contradictory required leap-second cases and the high-precision bignum cases, whose digits JavaScript numbers do not hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
Both evaluators take Python values, so the suite runner leaves out, by name, the high-precision bignum cases whose digits a float does not hold (and, for the Rust-backed one, the integer beyond 64 bits). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
As the crate's runner, it leaves out by name the high-precision bignum cases, whose digits PHP numbers and serde_json values do not hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
As the crate's runner, it leaves out by name the high-precision bignum cases, whose digits serde_json values do not hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
As the crate's runner, it leaves out by name the high-precision bignum cases, whose digits R's doubles and serde_json values do not hold. It passes once the package is built on crate 0.2.1 (checked locally with the crate patched in); on the crates.io 0.1.7 it pins, the format cases the crate fixed in 0.2.1 fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
… programs on the unified layout SuiteCommon.inc serves each case's externalSchemas and reads its compatibility and tags, and TestCompileSuite now includes it. The suite program expects 8,611 cases (less zero-terminated floats in draft4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
Both distributions leave out by name the high-precision bignum cases: the pure evaluator reads a number with a fraction as a double, and the XS one reads JSON through the C library's serde_json values; the Perl data comes from JSON::PP, which reads 2^64 as a double even with allow_bignum. The XS tests pass once Alien::CorvusJsonSchema takes C library 0.1.6, which has the format fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…on the unified layout The suite runner expects 8,611 cases (less zero-terminated floats in draft4); OCaml compares numbers exactly, so no bignum case is left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
As the C library's other runners, the suite test leaves out by name the high-precision bignum cases. It passes once CI downloads C library 0.1.6, the release with the format fixes (checked here with the library built from this branch). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
The exclusions were in the generator's appsettings.json, but the generated V5 tests still held the 28 cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…sions The conformance lines name the files left out (zero-terminated-floats.json, and the high-precision bignum cases for the engines whose values or serde_json numbers do not hold the digits), the skills and RunningTests.md describe the unified layout and the generator's settings, and the Bowtie READMEs the per-case registry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
The suite resolved our issue #1258 (json-schema-org/JSON-Schema-Test-Suite#1259): the required time tests expect a leap second to be refused for v1 only, so the six required leap-second cases the Rust, C library and TypeScript runners left out by name now run and pass. The new cases (formats, multipleOf, bignum) pass in every engine but three high-precision uniqueItems cases, which the engines that hold numbers as doubles (and V4) leave out by name, as they do the other bignum cases. The .NET suites are regenerated, and the counts in the runners and documents are the new ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
…1057) #1057 fixed the V4 dynamic scope for a $ref whose pointer reaches into an embedded resource, so the optional 2020-12 case is no longer excluded. The rebase also merged #1057's "skip a file that is not a test array" into the generator ahead of this branch's own check, which skipped every unified file (an object); the duplicate is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
.NET (V5 and V4), Rust and its C library, Go, R, Fortran and Perl XS now give a leap second the published rule, so their runners no longer leave the leap-second cases out: the generator configurations drop their 61 named exclusions, the runtime evaluator's runner its leap filter, and the .NET suite tests are regenerated. Java, Julia, OCaml, Pascal, pure Perl and pure Python still skip them until each engine has the rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3
mwadams
force-pushed
the
test/unified-suite
branch
from
November 11, 2026 20:07
c93b132 to
f1020ab
Compare
Contributor
Code Coverage Summary Report - Windows (net481)Summary
CoverageCorvus.Json.CodeGeneration - 78.6%
Corvus.Json.CodeGeneration.CSharp - 83.1%
Corvus.Json.ExtendedTypes - 78.9%
Corvus.Json.JsonReference - 73.6%
Corvus.Text.Json - 91%
Corvus.Text.Json.CodeGeneration - 64.2%
Corvus.Text.Json.JMESPath - 93.8%
Corvus.Text.Json.JMESPath.CodeGeneration - 97.7%
Corvus.Text.Json.Jsonata - 91.8%
Corvus.Text.Json.Jsonata.CodeGeneration - 88.6%
Corvus.Text.Json.JsonLogic - 95.1%
Corvus.Text.Json.JsonLogic.CodeGeneration - 92.8%
Corvus.Text.Json.JsonPath - 95%
Corvus.Text.Json.JsonPath.CodeGeneration - 96.1%
Corvus.Text.Json.Patch - 96.6%
Corvus.Text.Json.Toon - 88.2%
Corvus.Text.Json.Validator - 93.7%
Corvus.Text.Json.Yaml - 90.8%
Corvus.Toon.SystemTextJson - 90%
|
Contributor
Code Coverage Summary Report - Linux (net10.0)Summary
CoverageCorvus.Json.CodeGeneration - 80.3%
Corvus.Json.CodeGeneration.CSharp - 83.5%
Corvus.Json.ExtendedTypes - 80.8%
Corvus.Json.JsonReference - 76.4%
Corvus.Text.Json - 91.9%
Corvus.Text.Json.AsyncApi - 71%
Corvus.Text.Json.AsyncApi.Amqp - 0.2%
Corvus.Text.Json.AsyncApi.AzureServiceBus - 0%
Corvus.Text.Json.AsyncApi.CodeGeneration - 93.5%
Corvus.Text.Json.AsyncApi.Kafka - 0.2%
Corvus.Text.Json.AsyncApi.Mqtt - 0.4%
Corvus.Text.Json.AsyncApi.Nats - 0.2%
Corvus.Text.Json.AsyncApi.Polly - 57.8%
Corvus.Text.Json.AsyncApi.Testing - 96.2%
Corvus.Text.Json.AsyncApi.WebSocket - 0.6%
Corvus.Text.Json.CodeGeneration - 89.7%
Corvus.Text.Json.JMESPath - 93.8%
Corvus.Text.Json.JMESPath.CodeGeneration - 97.7%
Corvus.Text.Json.Jsonata - 91.6%
Corvus.Text.Json.Jsonata.CodeGeneration - 88.5%
Corvus.Text.Json.JsonLogic - 95.1%
Corvus.Text.Json.JsonLogic.CodeGeneration - 92.8%
Corvus.Text.Json.JsonPath - 94.7%
|
Contributor
Code Coverage Summary Report - Windows (net10.0)Summary
CoverageCorvus.Json.CodeGeneration - 80.3%
Corvus.Json.CodeGeneration.CSharp - 83.5%
Corvus.Json.ExtendedTypes - 80.8%
Corvus.Json.JsonReference - 76.6%
Corvus.Text.Json - 91.9%
Corvus.Text.Json.AsyncApi - 71%
Corvus.Text.Json.AsyncApi.Amqp - 0.2%
Corvus.Text.Json.AsyncApi.AzureServiceBus - 0%
Corvus.Text.Json.AsyncApi.CodeGeneration - 93.5%
Corvus.Text.Json.AsyncApi.Kafka - 0.2%
Corvus.Text.Json.AsyncApi.Mqtt - 0.4%
Corvus.Text.Json.AsyncApi.Nats - 0.2%
Corvus.Text.Json.AsyncApi.Polly - 57.8%
Corvus.Text.Json.AsyncApi.Testing - 96.2%
Corvus.Text.Json.AsyncApi.WebSocket - 0.6%
Corvus.Text.Json.CodeGeneration - 89.7%
Corvus.Text.Json.JMESPath - 93.8%
Corvus.Text.Json.JMESPath.CodeGeneration - 97.7%
Corvus.Text.Json.Jsonata - 91.6%
Corvus.Text.Json.Jsonata.CodeGeneration - 88.5%
Corvus.Text.Json.JsonLogic - 95.1%
Corvus.Text.Json.JsonLogic.CodeGeneration - 92.8%
Corvus.Text.Json.JsonPath - 94.7%
|
mwadams
marked this pull request as ready for review
November 11, 2026 22:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every JSON-Schema-Test-Suite runner in the repository moves to the suite's unified layout. The submodule is pinned to 6a23a51, the suite's
mainnow that the unified layout has landed there.The unified layout
validation/tests, for every dialect. Each test case has acompatibilityrange of releases (4, 6, 7, 2019, 2020, 9999 for v1), and the runners apply it with the README's algorithm.tagssay whether a case expectsformatto be asserted (format-assertion) or only annotated (format-annotation).externalSchemas, scoped to that case, in place of the sharedremotes/directory (the same URI can have different bodies in different cases).annotations/testsis almost unchanged;output-testsis gone.proposals/are out of scope here.What changes
SuiteRunner, the evaluator test suite, the tests that readremotes/, and the V4 test generator and its generated tests (V4 is ported too, as the docs say it is supported going forward).No engine changes are needed: the format, UTS #46 and exact-number fixes shipped first (#1053, V5.8.3 to V5.8.4 and the port releases), the notices in #1056, and the V4 dynamic-scope fix in #1057 (V5.8.5), whose exclusion this PR drops.
Exclusions that remain, by name
format. Each engine's leap-second release removes its skip. .NET (V5 and V4), Rust and its C library, Go, R, Fortran and Perl XS have the rule (V5.8.6, crate 0.2.2, Go 0.1.5 and the bindings built on them), so their runners now run every leap-second case and the 61 named .NET exclusions are gone.Decimal).Totals on 6a23a51
Local gate after rebasing on main (4961cdc): solution build 0 warnings; 99,801 .NET tests passed (net10.0), 0 failed; analyzer configuration and code-sample catalog current. Run locally on the same commit: Go 8,686, Rust crate and C API 8,638, R (
R CMD checkOK), Fortran and Perl XS 8,633, annotations 213 in each.🤖 Generated with Claude Code
https://claude.ai/code/session_011rjLiHWYWRFfJb3dkSsnE3