Colander and None during deserialization?

[tsauerwein]

I am confused about the way Colander handles None values (or not handles None values) during the deserialization. For example:

class Person(colander.MappingSchema):
    name = colander.SchemaNode(colander.String(), missing=None)

schema = Person()
print(schema.deserialize({'name': None}))
# {'name': None}

In this case Colander accepts a None value, and keeps it because missing is set to None. This sounds reasonable to me.

Another example using a sequence:

class NoneSequence(colander.MappingSchema):
    foo = colander.SchemaNode(colander.String())
    bar = colander.SchemaNode(
        colander.Sequence(), colander.SchemaNode(colander.String()),
        missing=None)

schema = NoneSequence()
deserialized = schema.deserialize({'foo': '1', 'bar': None})
# colander.Invalid: {'bar': '"None" is not iterable'}

In this case Colander does not accept None.

Yet another example using a nested schema:

class SchemaA(colander.MappingSchema):
    val = colander.SchemaNode(colander.String())


class SchemaB(colander.MappingSchema):
    a = SchemaA(missing=None)


class SchemaC(colander.MappingSchema):
    b = SchemaB()

schema = SchemaC()
print(schema.deserialize({"b": {"a": None}}))
# colander.Invalid: {'b.a': '"None" is not a mapping type: Does not implement dict-like functionality.'}

Again None is not accepted. I have some more examples with nested sequences and tuples here: https://gist.github.com/tsauerwein/93c88ba3c37f936dea113a27447c0db5

What is the idea? Why does Colander accept None in some cases during the deserialization, and in some cases not? Do I have to replace all None values with colander.null before calling deserialize?

Looking the tests it seems that this is even the expected behavior:
https://github.com/Pylons/colander/blob/cb1645d/colander/tests/test_colander.py#L935
https://github.com/Pylons/colander/blob/cb1645d/colander/tests/test_colander.py#L690

[weilu]

I ran into the same issue. Any explanation would be great!

[delijati]

There was already a solution to that #258

[ngoonee]

It wasn't merged though. Does that mean I have to search-and-replaces None values with colander.null currently?

[tsauerwein]

Does that mean I have to search-and-replaces None values with colander.null currently?

That's what we ended up doing.

[ngoonee]

Ah, that's what I was afraid of. Not hard to do but I'm dreading having to remember to come back to this 2 years from now....

[delijati]

in #258 a contributor had to sign and did not do it ...

[ngoonee]

I understand, not trying to apportion any blame. Stuff happens.

[leplatrem]

I would be willing to contribute a PR. But before starting, which approach would the maintainers recommend to solve the None deserialization issue? Same as #258 or a recursive replace of None to colander.null before deserializing?

In #258 @mmerickel mentioned the allow_empty option. Currently, it only applies to the String schema type. Should we generalize it to all of them?

[RangelReale]

I'm having this exact same problem, I have a SequenceSchema that is optional, and I use the missing parameter. If I don't send the field in the json, it works, but if I send null (None), I get the '"None" is not iterable' message.

[stevepiercy]

@mmerickel is redoing the work in #258 the preferred way forward to resolve this issue for non-strings? Per your comment, strings are already handled via allow_empty.

If that's all it takes, then I am happy to do the copy-paste from the code in the PR #258 to get this into Colander.

[mmerickel]

#258 wouldn't work here and looking at its impl again I wouldn't accept it as-is. It's converting none explicitly to an empty string and in the case of a SequenceSchema you'd want it to deserialize to something else.

WRT some other comments in this thread, I can't actually explain the scenarios in which colander.null is intended to be used as I was not the original developer of this library and have never needed to use it myself. The semantics around it are pretty confusing to me.

@RangelReale None and "missing" are different concepts entirely in schema design, even if you would prefer them to be treated the same in your case. If you want to handle None, I would suggest using a preparer to delete the value entirely. At which point the "missing" semantics will kick in and you can specify a default value there such as colander.drop or None or anything else.

[stevepiercy]

Thanks for the quick reply @mmerickel. Also after reviewing your earlier comment about the proposed implementation treating "" as None and vice versa, I would agree that the proposal in #258 needs to be modified further. I missed that the first time through.

For this issue, I would suggest that developers carefully review the documentation of The Null and Drop Values. This is densely packed with easy to misunderstand details. There are also two tables which illustrate Serialization Combinations and Deserialization Combinations.

I think this issue can be closed, as it seems to be a matter of not understanding how Colander handles None during deserialization, which I think is answered and resolved in the documentation I mention above. Please do speak up if that is not accurate.

[RichardJones1337]

I'm also getting this issue with nested schema and handling 'null' values.
Say I have the schema:

class SchemaA(colander.MappingSchema):
    val = colander.SchemaNode(colander.String())

class SchemaB(colander.MappingSchema):
    a = SchemaA(missing=None)
    otherval = colander.SchemaNode(colander.String(), missing=None)

deserialising the json:

{
    "a": null,
    "otherval": "asdf"
}

gives the error:

{'a': '"None" is not a mapping type: Does not implement dict-like functionality.'}

based on the documentation:

If the appstruct value computed by the type's deserialize method is colander.null and the schema node has an explicit missing attribute (the node's constructor was supplied with an explicit missing argument), the missing value will be returned. Note that when this happens, the missing value is not validated by any schema node validator: it is simply returned.

I would expect a to be deserialised to None