CRITICAL: HED validator never reports errors (severity enum vs string bug)

[neuromechanist]

Severity

Critical. The Python HED validator classifies every issue as a warning and reports is_valid = True for HED strings that contain real TAG_INVALID (and any other error-severity) issues. Validation is effectively a no-op for errors, and the annotation workflow's refine loop never triggers on invalid tags.

Root cause

src/validation/hed_validator.py, HedPythonValidator.validate (around lines 170-186):

severity: Literal["error", "warning"] = (
    "error" if issue["severity"] == "error" else "warning"   # <-- always False
)
...
if severity == "error":
    errors.append(validation_issue)
else:
    warnings.append(validation_issue)
is_valid = len(errors) == 0    # <-- always True

hedtools (hed 1.1.1) sets issue["severity"] to an hed.errors.error_types.ErrorSeverity IntEnum (ErrorSeverity.ERROR == 1, ErrorSeverity.WARNING == 10), not the string "error". So issue["severity"] == "error" is always False, every issue is bucketed as a warning, errors is always empty, and is_valid is always True.

Reproduce:

from hed import HedString, load_schema_version
from hed.validator import HedValidator
from hed.errors.error_types import ErrorSeverity
s = load_schema_version("8.4.0"); v = HedValidator(s)
issues = HedString("Sensory-event, Sky", s).validate(v)
print(issues[0]["severity"])                       # ErrorSeverity.ERROR (1)
print(issues[0]["severity"] == "error")            # False  <-- the bug
print(issues[0]["severity"] == ErrorSeverity.ERROR) # True

Impact

Real-world: a downstream 1000-image annotation run (with the new extension policy) produced HED where only 99/1000 strings are actually valid; hedit reported 1000/1000 valid. 2,927 TAG_INVALID errors (bare non-schema tags like Sky, Water, Metal, Wood used directly instead of a real parent + extension) all passed silently. The workflow's max_validation_attempts refine loop never fired because no error was ever seen.

This stayed hidden until extensions were used (bare/extended tags are the main error source); pre-extension annotations that only used in-schema tags happened not to trip it.

Fix

Compare against the enum (and treat unknown as error, fail-closed):

from hed.errors.error_types import ErrorSeverity
sev = issue.get("severity")
is_error = sev == ErrorSeverity.ERROR or sev == int(ErrorSeverity.ERROR)
severity = "error" if is_error else "warning"

Also add a regression test with a known-invalid string (e.g. "Sensory-event, Sky") asserting is_valid is False, and one with a valid extension ("Natural-feature/Sky") asserting is_valid is True. Check the JS validator path for the analogous classification too.

[VisLab]

@neuromechanist: will try to release 1.1.2 later today. We've isolated the issue.

[VisLab]

@neuromechanist We believe that it is a documentation error rather than a code error: Can you fix the HEDit validation code to compare against ErrorSeverity enum instead of strings. If this works, then we will delay a patch release for a few days and possibly want until a minor release.

[neuromechanist]

Thanks for the follow up, @VisLab, I did not know that you follow this repo (I think you follow every repo).
Yes, will do.
This came up as we are experimenting with a limited set of tag extensions that would enrich the image annotation, quite significantly.