[Clarification Needed] Validation and code generation of open-ended "type: object"

[sebastien-rosset]

Bug Report Checklist

• Have you provided a full/minimal spec to reproduce the issue?
• Have you validated the input using an OpenAPI validator (example)?
• What's the version of OpenAPI Generator used?
• Have you search for related issues/PRs?
• What's the actual output vs expected output?
• [Optional] Bounty to sponsor the fix (example)

Description

According to the JSON schema spec, when a schema is an open-ended "type: object", such as below, the document must be an unordered set of properties mapping a string to an instance, from the JSON "object" production.

components:
  schemas:
    Blob:
      type: object

In the above example, additionalProperties is not specified, hence it has the default value "true"

AFAIK, that means the document cannot be the "null" value, it cannot be a primitive type (e.g. string, integer...) and it cannot be an array. Is my understanding correct?
If on the other hand, the intent is the document can take any valid JSON value, then with the JSON schema specification, and putting the OAS spec aside for a moment, it could be specified with a type array:

type: [ 'null', object, array, boolean, string, number ]

But type arrays are not supported in OAS, so instead my question is are we supposed to use "oneOf"?

components:
  schemas:
    Blob:
      nullable: true      # cannot use type: 'null' in OAS 3.0
      oneOf:
        - type: object
        - type: array
        - type: boolean
        - type: string
        - type: number

But in practice, I see multiple generators are lenient for "type: object" and accept any valid JSON document. Even if we use "oneOf" in the spec, multiple generators do not support 'oneOf' very well, so I am reluctant to use that construct for this specific purpose (because I need to generate SDK that work for multiple languages). I haven't gone through the entire list, but I see at least two different behaviors across two generators. For example, "python-experimental" accepts any JSON valid document:

'blob': (bool, date, datetime, dict, float, int, list, str, none_type,),  # noqa: E501

But with the same OAS spec, go-experimental implements a map[string]interface{}, which means the value can either be the nil value or a map of string to any value; with strict validation, it would have to be written as:

type: [ 'null', object ]

openapi-generator version

master February 20th 2020

OpenAPI declaration file content or url

Command line used for generation

Steps to reproduce

Related issues/PRs

Suggest a fix

[sebastien-rosset]

In this specific scenario, I think the OAS spec is clear (but maybe I missed one crucial sentence). I.e. "type: object" can only a map from string to arbitrary values.

In OpenAPITools, there are fields and methods named "FreeForm". That term is not used anywhere in the JSON schema spec, and it is used a couple times in the OAS spec. I think it conveys semantically that it can be any arbitrary type, not just a map where the keys are string.

Recently I added documentation to ModelUtils.isFreeFormObject(), but I think it's not sufficient. I'd like to clarify. https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/java/org/openapitools/codegen/utils/ModelUtils.java#L671

One possible clarification could be 1) "the value of a Free Form schema can be any valid value (e.g. any valid JSON or XML document)". Or 2) "the value of a Free Form schema is an unordered set of properties mapping a string to any valid value". Unfortunately, some code generators interpret with definition 1, others with definition 2

[richardwhiuk]

Regarding type arrays, yes I think the expectation would be that you use oneOf.

[sebastien-rosset]

Regarding type arrays, yes I think the expectation would be that you use oneOf.

So if we go with the approach that "type: object" is (exclusively) an unordered set of properties mapping a string to any valid value, the following would have to happen:

1. Fix existing code generators to apply strict validation, as specified in the spec.
2. But (1) will probably make a lot of people unhappy (depending on the specific code generator). In that case should we add a configuration parameter to control how to interpret "type: object" at codegen time, or possibly a runtime option?
3. OAS authors change their spec to use oneOf, which will take a lot of time, or may never happen
4. Propose that OAS spec > 3.0 supports type arrays, in order to have more concise OAS document.

Also, it would be really nice if the generated code oneOf code has special cases such that the generated code is not ugly.

[spacether]

How is the go interface behavior different than the python-experimental implementation? Can you provide an example? I am not following you from your explanation.
python-experimental requires that the key be a string and that the values be of any type when {} is used.

[sebastien-rosset]

How is the go interface behavior different than the python-experimental implementation? Can you provide an example? I am not following you from your explanation.
python-experimental requires that the key be a string and that the values be of any type when {} is used.

Given the following schema:

components:
  schemas:
    blob:
      type: object
      properties:
         stuff:
            type: object

The generated go-experimental client will accept the following JSON payloads:

"stuff": { "prop-1": true, "prop-2": "abc", "prop-3": 123.5, "prop-4": [1, "abc"] }
"stuff": null

But the following payloads will fail:

"stuff": 123
"stuff": [ "abc", 123 ]
"stuff": true
"stuff": 123.454

AFAIK, the generated python-experimental client will accept all of the above.

My understanding of the spec is that neither the current go and python implementations are correct in interpreting the OAS specification (but as mentioned in the referenced issue, it's possible I have missed one important sentence). Though go is the closest match. On the other hand, I like the leniency of the Python implementation, and I am guessing a significant number of server-side implementations interpret "type: object" as a truly free-form valid JSON document.

[spacether]

Ah, thank you for explaining that. How about we switch python-experimental to definition #2?

[sebastien-rosset]

Ah, thank you for explaining that. How about we switch python-experimental to definition #2?

The problem is I wouldn't be surprised if a bunch of OAS authors use definition 1. Do you think most authors care to write the mouthful schema for arbitrary JSON documents:

components:
  schemas:
    Blob:
      nullable: true      # cannot use type: 'null' in OAS 3.0
      oneOf:
        - type: object
        - type: array
        - type: boolean
        - type: string
        - type: number

One practical issue I see is that many validators are lenient, the OAS spec lacks examples, so it's easy to assume that "type: object" equates arbitrary JSON document.

In that case, the python-experimental SDK would not work for them. Maybe it could be a build-time configuration option, as I proposed?

Also, it's awful if the generated methods end up being something like GetOneOfObjectOrArrayOrBooleanOrStringOrNumber(). Then the author swaps elements in the array, and the generated code becomes GetOneOfArrayOrObjectOrBooleanOrStringOrNumber. Ouch.

In that case in golang it would be nice to have just GetBlob() interface{}. But that's a special case. On the other hand the schema could also use referenced, structured schemas.

    Blob:
      oneOf:
        - type: '#/components/schema/Car'
        - type: '#/components/schema/Building'

[spacether]

Have any other generators implemented the {} support? Also, OneOf does seem like the cleaner way to explicitly define types of a parameter.
If the generators do not yet support it, that means that we should build more.

[sebastien-rosset]

Have any other generators implemented the {} support? Also, OneOf does seem like the cleaner way to explicitly define types of a parameter.
If the generators do not yet support it, that means that we should build more.

The Java client does, but otherwise I don't know.

[spacether]

Did java do it the map way or the any type way?

[sebastien-rosset]

Did java do it the map way or the any type way?

I don't know enough about the Java generator, but it looks like it uses generics:

 org.openapitools.jackson.nullable.JsonNullable<Object>

https://github.com/OpenAPITools/jackson-databind-nullable/blob/master/src/main/java/org/openapitools/jackson/nullable/JsonNullable.java

[richardwhiuk]

I've dug into this a bit to decide what to do in the Rust Server generator.

1. No type specified at all:

components:
  schemas:
    Blob:
      nullable: true

This is allowed to be anything - and is thus equivalent to the anyOf solution proposed further up.

2. Specifying the type as object.

components:
  schemas:
    Blob:
      nullable: true
      type: object

which makes it a requirement that the object contains properties.

ModelUtils.isFreeFormObject() - https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/java/org/openapitools/codegen/utils/ModelUtils.java#L671-L708

currently treats these two cases as identical.