Multi platform container

[sladage]

From what I understood from the specs is that a container targets a single platform? If so, wouldn't it be better to allow for different configurations depending on the platform, all packaged into one container?

[wking]

On Thu, Jul 23, 2015 at 10:12:43AM -0700, Stefan Ladage wrote:

From what I understood from the specs is that a container targets a
single platform? If so, wouldn't it be better to allow for different
configurations depending on the platform, all packaged into one
container?

I was recently confused about this too, but I think the current spec
almost allows this already. We just need to remove the requirement
for the config.json name 1. The ‘runc’ command already supports
launching a container from an alternative spec 2, although I haven't
been able to find clear docs describing this functionality.

However, this still requires the launcher to figure out which of
several configs to use for their platform. I think it might make more
sense to replace the default config.json with a config/ directory.
The launcher (e.g. runC) would walk that directory to find potential
config files (anything ending in ‘.json’?) and then figure out which
of the found spec files was the best platform match (erroring out if
there were no compatible specs).

To keep the config DRY, it would be nice if there was also a way to
share code between configs. For example, inheritance:

{
…
"parents": [
"../base.json",
"./amd64.json"
],
…
}

or embedding (e.g. with JSON Schema's $ref 3:

{
…
"linux": {
"$ref": "./amd64.json#linux"
}
…
}

Since inheritance and embedding are potentially complicated, I'm fine
punting on those for now, but I do think that a walkable config
directory is a clear win for multi-platform fat containers.

Of course, keeping the option to manually specify a config file when
invoking runC is good too. For example, you may have one config to
launch your indended application, and another with something like:

{
"version": "pre-draft",
"parents": ["./main.json"],
"process": {
"args": ["bash"]
}
}

that's identical to the app config but just launches a shell. For the
default platform-matching logic, you'd want a way to break ties like
this. I'd suggest either C-locale sorts of the relative path from
config/ (earlier paths win) or an explicit “"priority": ”
setting (higher priorities win).

[sladage]

Why not change the platform section into something like this:

"platforms":[
{
    "os": "linux",
    "arch": "amd64",
    "config":"linux64.json"
},
{
    "os": "windows",
    "arch": "amd64",
    "config":"win64.json"
}
]

The main config file serves as a base and the platform specific configs may override properties from the base config or add platform specific sections.

[wking]

On Fri, Jul 24, 2015 at 04:27:55AM -0700, Stefan Ladage wrote:

Why not change the platform section into something like this:

"platforms":[
{
    "os": "linux",
    "arch": "amd64",
    "config":"linux64.json"
},
{
    "os": "windows",
    "arch": "amd64",
    "config":"win64.json"
},
]

The main config file serves as a base and the platform specific
configs may override properties from the base config or add platform
specific sections.

That saves a directory walk vs 1, and it shifts the platform
information from linux64.json to the parent config's platform entry.
But it doesn't give you $ref to share arbitrary snippets [1,2].

Do you expect it to be recursive? For example, could linux.json have
(following #39):

"platforms":[
{
"os": "linux",
"architecture": "arm64",
"variant": "aarch64"
"endian": "big",
"abi": "ipl32"
"config": "aarch64-be-ipl32.json",
},
{
"os": "linux",
"architecture": "arm64",
"variant": "aarch64"
"endian": "little",
"abi": "ipl32"
"config": "aarch64-le-ipl32.json",
},
…
]

I'm not sure which way (this or 1) would be easier to use. Say you
have N platforms, M entry-points, …. How do we avoid forcing authors
to write N⋅M⋅… configs, while still giving them the freedom to
customize a given combination if they need to? With your explicit
child approach, you could do things like a single shell config for all
platforms:

{
"version": "pre-draft",
"platforms": [ {"os": "all"} ],
"process": {"args": ["bash"]}
}

(os no need for N⋅M⋅… configs), but I'm not sure how you'd tweak the
arguments for aarch64-le-ipl32 (where maybe you couldn't get Bash to
compile and wanted to use Dash).

With my explicit parent approach 1, you need an override config for
every leaf in the platform tree
(e.g. config/linux/aarch64/le/ipl32-shell.json, …). You could use
$ref to link to the override itself (in case it was more involved than
a single setting):

{
"version": "pre-draft",
"parents": ["./ipl32-main.json"],
"$ref": "../../../dash.json"
}

but it would still be tedious.

One approach would be to add if/then logic and such to the config, but
dynamic configs are no fun (which is why setup.py can be such a pain
in Python).

I think the best choice at the moment is:

1. If config.json exists (or you were passed a file path), use that.
2. If config/ exists (or you were passed a directory path), walk it
   looking for the best platform match and use that.

And then just punting on fancy inheritance ("$ref", "config",
"parents", …) until someone has a better idea than we've had so far
;).

[philips]

Can someone write-out a use case for this? What is an example use case for a bundle that supports multiple archs? Why is having two bundles too complicated?

[crosbymichael]

I don't think this is wise. A config + rootfs probably cannot be amd64 and win64 at the same time. You will have to have two different configs.

[glevand]

This single multi platform container idea adds complexity. I can't think of how it would have an advantage over multiple single platform containers. Is there a real need?

[wking]

On Tue, Jul 28, 2015 at 12:30:16PM -0700, Geoff Levand wrote:

This single multi platform container idea adds complexity. I can't
think of how it would have an advantage over multiple single
platform containers.

Somebody is going to have to decide which of a family of $X is the
best match for a given platform. Whether that's picking the best
config from a given directory (#76) or picking the best bundle from a
set of related bundles doesn't really matter to me. But I'll want a
way to say “I'd like to launch an Nginx container” and have a tooling
layer underneath me that setup a container appropriate to whatever
hardware I was on. I'm fine pushing this to a higher-level than this
spec, but I'm not sure where that higher level is. See also 1,
where I wonder about how we expect coordination with higher (or
parallel) levels to work.

[sladage]

Having multi-platform containers allow you to deploy the same containers on multiple machines with different platforms, greatly simplifying the deployment process. Isn't the whole idea of using containers that you don't have to worry about what your deployment target looks like?

As for the added complexity, the complexity of having to deal with multiple platforms is there whether we deal with it here, in this spec, or make it someone else's problem. At least if we add it to this spec, we have standardized way of deploying to multiple platforms.

Personally, I can see a lot of benefits using a single container with shared resources which I can move between linux, osx and windows.

[stevvooe]

While this is a real use case, for the current level of runc, this is out of scope.

We can imagine a distribution system that accomplishes this task, coordinating the cross-platform creation of bundles. This is not part of the near term goals.

[wking]

On Thu, Jul 30, 2015 at 11:11:29AM -0700, Stephen Day wrote:

While this is a real use case, for the current level of runc, this
is out of scope. We can imagine a distribution system that
accomplishes this task, coordinating the cross-platform creation of
bundles.

That's a reasonable approach. Does anyone know of a spec for that
higher-level system? One approach is 1, but that's just me ;). I
imagine the CNCF folks are interested in this area, but I can't find
links to a spec on their site 2. Does anyone here know anyone
involved with the CNCF or Kubernetes? Do we need a new project to
discuss these higher levels?