Date regex not updated to fixed (v1.04) version in json schema, or in main branch

[fsuits]

Describe the bug

OSCAL v1.0.4 included a fix to the regex for dates but the json schema for catalog and possibly others does not parse all dates properly. The regex in the json schema does not appear to have ever changed from v1.0.0

The fix is described in the changelog for v1.0.4 which refers to the bug report #1260

The regex appears twice around line 900 in the catalog schema for published and last-modified
https://github.com/usnistgov/OSCAL/blob/v1.0.4/json/schema/oscal_catalog_schema.json

If you compare the regex near line 900 for published or last-modified it does not appear to have changed in v1.0.4 or main branch.

Who is the bug affecting

Developers who rely on the json schema and users of code based on that schema

What is affected by this bug

OSCAL Content

How do we replicate this issue

See the regex pattern near line 900 of https://github.com/usnistgov/OSCAL/blob/v1.0.4/json/schema/oscal_catalog_schema.json

^((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$

See that the regex will only parse the date part of 2022-05-03T02:00:00Z

But it will parse the entire date/time for 2022-06-03T02:00:00Z

using a parser such as https://www.debuggex.com/

Expected behavior (i.e. solution)

The regex should be updated from v1.0.0 in the json schema so it parses all valid date/time strings properly.

Other comments

No response

[Compton-US]

For when this is reviewed, it looks like this shows up in multiple files in the repo, beyond the catalog.

find . -type f -name "*.*" -exec egrep -Hn -A0 -B0 "\"\^\(\(200" {} ";" | egrep "\)T\(" | cut -d ":" -f 1 | sort | uniq -c

Result:

   2 ./build/metaschema/test-suite/schema-generation/datatypes/datatypes-datetime-no-tz_json-schema.json
   2 ./build/metaschema/test-suite/schema-generation/datatypes/datatypes-datetime_json-schema.json
   4 ./build/metaschema/toolchains/xslt-M4/schema-gen/oscal-auto-datatypes.xsd
  10 ./json/schema/oscal_assessment-plan_schema.json
  14 ./json/schema/oscal_assessment-results_schema.json
   2 ./json/schema/oscal_catalog_schema.json
  14 ./json/schema/oscal_complete_schema.json
   2 ./json/schema/oscal_component_schema.json
  10 ./json/schema/oscal_poam_schema.json
   2 ./json/schema/oscal_profile_schema.json
   2 ./json/schema/oscal_ssp_schema.json

Could this be in metaschema?

• https://github.com/usnistgov/metaschema/blob/main/toolchains/xslt-M4/schema-gen/oscal-auto-datatypes.xsd#L215

   <xs:simpleType name="dateTime-with-timezone">
      <xs:restriction base="xs:string">
         <xs:whiteSpace value="preserve"/>
         <!-- type: string -->
         <!-- format: date-time -->
         <xs:pattern value="^((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$"/>
      </xs:restriction>
   </xs:simpleType>

• https://github.com/usnistgov/metaschema/blob/main/toolchains/xslt-M4/schema-gen/make-json-schema-metamap.xsl#L665

        <map key="dateTime-with-timezone"><!-- DateTimeWithTimezoneDatatype -->
            <string key="type">string</string>
            <string key="format">date-time</string>
            <string key="pattern">^((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$</string>
        </map>

[david-waltermire]

The Metaschema main branch has not been updated with this change yet, but develop has it.

OSCAL is pointing to the correct Metaschema submodule commit, which is in the develop commit history.

The actual problem is the XML and JSON schemas were not regenerated for the v1.0.4 release. I believe the CI/CD has not regenerated the XML and JSON schema files. A good way to fix this is to release a v1.0.5 with the correct schemas once the CI is confirmed to be working.

[aj-stein-nist]

The Metaschema main branch has not been updated with this change yet, but develop has it.

OSCAL is pointing to the correct Metaschema submodule commit, which is in the develop commit history.

The actual problem is the XML and JSON schemas were not regenerated for the v1.0.4 release. I believe the CI/CD has not regenerated the XML and JSON schema files. A good way to fix this is to release a v1.0.5 with the correct schemas once the CI is confirmed to be working.

This is very good to know. Thanks for the catch, @david-waltermire-nist. I will quickly investigate the release and schema updates issue and factor in how we can work on this in the next sprint (Sprint 65) or a following one.

[aj-stein-nist]

For Sprint 65 @aj-stein-nist is the driver for this issue.

[aj-stein-nist]

@JustKuzya will find some time to pair with me and we begin work on this.

[aj-stein-nist]

OK @fsuits thanks for confirming. @JustKuzya and I got around to confirming what @david-waltermire-nist is the case and we understand a few approaches to resolving this, each with different levels of effort. More to follow!

[aj-stein-nist]

@JustKuzya and I are going to work with Dave on the Metaschema to try and hotfix the usnistgov/metaschema#229 for usnistgov/metaschema#224 to a main branch like what was used for v1.0.4 (instead of all changes inclusive from then current main of metaschema repo during v1.0.4 release of OSCAL to current develop commit) in usnistgov/metaschema#341. Replaying commits appears to not be sufficient, as not just file relocation occurred, but file renaming. If we cannot do it with this low-lift approach, we will move this into the next sprint and adjust accordingly.

[aj-stein-nist]

After consultation with Dave last night, it seems we may move up to develop sub-module and test that first, or we will have to opt for messier git surgery with the cherry-picking than we thought. More to follow.

[aj-stein-nist]

I am driving this one so I owe a status update: Dmitry and I worked on it but I held it up working on other issues before we move up the module. Working #1726 means plotting a way forward and acting on it is hard until that is out of the way (to re-run the reference docs and website model pieces will fail, and are very relevant to this work). More to follow!