Construction of OmegaConf from Structured Configs

[omry]

The associated PR (#86 ) is adding support for initializing OmegaConf from class files and objects.
Details below

1. Config classes and objects (dataclass or attrs classes) can be converted to OmegaConf object.

2. The following value types are supported:

• with a default value assigned to it.
• Optional fields (can accept None)
• assigned omegaconf.MISSING, Equivalent for '???' in the yaml files (Mandatory missing).
  The semantics here is that this value is not yet set. it must be set prior to read access.
• Interpolation using omegaconf.II() function :
  II("foo") is equivalent to ${foo} in the YAML file.
  Examples:
  • foo=II("bar") : Foo inherits the type and value of bar at runtime.
  • foo=II("http://${host}:${port}/") : foo is always a string. actual value is determined by host and port
  • foo=II("env:USER") : foo is evaluated on access, the type and value is determined by the function (env in this case)

Example for bool types (The same is supported for int, str, float and enum):

@dataclass
class BoolConfig:
    with_default: bool = True
    null_default: Optional[bool] = None
    mandatory_missing: bool = MISSING
    interpolation: bool = II("with_default")

3. the typing information in them is available and acted on at runtime (composition and command line overrides are at runtime).

4. type annotations can be used to get static type checking.

Currently the following is supported for both dataclasses and attr classes.
Examples below are @dataclasses. attr classes are similar.

@dataclass
class Database:
    name: str = "demo_app_db"
    # A an untyped list. will accept any type that can be represented by OmegaConf
    tables: List = field(default_factory=lambda: ["table1", "table2"])
    port: int = 3306

    # A list of integers, will reject anything that cannot be converted to int at runtime
    admin_ports: List[int] = field(default_factory=lambda: [8080, 1080])

    # A dict of string -> int, will reject any value that cannot be converted to int at runtime
    error_levels: Dict[str, int] = field(
        default_factory=lambda: {"info": 0, "error": 10}
    )

This class, and objects of this class can be used to construct an OmegaConf object:

# Create from a class
conf1 = OmegaConf.create(Database)

# Create from an object
conf2 = OmegaConf.create(Database(port=3307))

Python type annotation can then be used on the config object to static type checking:

def foo(cfg : Database):
   cfg.port = 10 # passes static type check
   cfg.port = "nope" # fails static type check

5. Composition and overrides are often happening at runtime (from files, or command line flags).
   To support that, there is also a runtime validation and conversion.
   The following will succeed at runtime:

conf.merge_with_dotlist(['port=30']) # success
conf.merge_with_dotlist(['port=nope']) # runtime error

6. OmegaConf objects created from Structured configs will be set to struct mode, this means that any access to a field not in the struct will result in a runtime error:

conf.pork = 80 # fail

7. Typed containers are validated at runtime:

conf: Database = OmegaConf.create(Database)
conf.admin_ports[0] = 999  # okay
assert conf.admin_ports[0] == 999

conf.admin_ports[0] = "1000"  # also ok!
assert conf.admin_ports[0] == 1000

with pytest.raises(ValidationError):
    conf.admin_ports[0] = "fail"

with pytest.raises(ValidationError):
    conf.error_levels["info"] = "fail"

8. Nested Object/Classes are supported, as well as Enums.

class Protocol(Enum):
    HTTP = 0
    HTTPS = 1


@dataclass
class ServerConfig:
    # Nested configs are by value.
    # This will be expanded to default values for Database
    db1: Database = Database()

    # This will be expanded to the values in the created Database instance
    db2: Database = Database(tables=["table3", "table4"])

    host: str = "localhost"
    port: int = 80
    website: str = MISSING
    protocol: Protocol = MISSING
    debug: bool = False

9. Enums can be assigned by Enum, by name string or by value (the later should be used at runtime only).

    conf: ServerConfig = OmegaConf.create(ServerConfig(protocol=Protocol.HTTP))
    assert conf.protocol == Protocol.HTTP

    # The HTTPS string is converted to Protocol.HTTPS per the enum name
    conf.protocol = "HTTPS"
    assert conf.protocol == Protocol.HTTPS

    # The value 0 is converted to Protocol.HTTP per the enum value
    conf.protocol = 0
    assert conf.protocol == Protocol.HTTP

    # Enum names are care sensitive
    with pytest.raises(ValidationError):
        conf.protocol = "https"
 
    for value in (True, False, "1", "http", 1.0):
        with pytest.raises(ValidationError):
            conf.protocol = value

10. Frozen state is preserved (and is recursive):

    @dataclass(frozen=True)
    class FrozenClass:
        x: int = 10
        list: List = field(default_factory=lambda: [1, 2, 3])

    conf = OmegaConf.create(FrozenClass)
    with pytest.raises(ReadonlyConfigError):
        conf.x = 20

    # Read-only flag is recursive
    with pytest.raises(ReadonlyConfigError):
        conf.list[0] = 20

[shagunsodhani]

Regarding:

OmegaConf objects created from Structured configs will be set to struct mode, this means that any access to a field not in the struct will result in a runtime error:
conf.pork = 80 # fail

Maybe the user could pass an argument to decide wether the object should be created in the strict mode or not.

[omry]

@shagunsodhani, thanks for the suggestion.
I think using a data class implies closed structure. you can always change the flag with
OmegaConf.set_struct(config_node, False).

This brings up the use case of creating a config from a typed Dict or List:

d : Dict(str, int) = {"foo" : 10}
cfg = OmegaConf.create(d)

This should retain the types but it currently doesn't.

[omry]

Need to check if this is even technically possibly, it's possible that without a class the annotations does not actually exists at runtime.

[omry]

@shagunsodhani, can't be done as I wrote it.
You should be able to use Dict inside your struct and it will not be in struct mode.

[shagunsodhani]

Sorry @omry I missed this notification. I think you are right. There are some libraries to get the type from functions/classes but for variables, they only way I could find was to use __annotations__ in global or local.