Proposal to add a user layer to the library

[Emdot]

This is to start a discussion about a possible enhancement to the project.

The library already does a great job of exposing the Docker REST API to .NET consumers on a 1:1 basis of .NET methods to REST calls, and doing all the heavy lifting for transport and streaming and such. I think the next step is to provide a layer on top of that excellent foundation, to expose the functionality in a way that's more idiomatically .NET. The goal would be to provide a low barrier to entry to access Docker via .NET--even for people who are new to Docker.

Three examples to get us started:

1. ContainerOperations.AttachContainerAsync needs to know whether the container was started with TTY. It has no way to get that without either doing an extra REST call (which doesn't fit the 1:1 mapping philosophy) or getting it from the caller (which puts a burden on the caller). In the proposed layer, the AttachAsync would use two calls to the lower layer--one to get the TTY value and one to ContainerOperations.AttachContainerAsync.
2. ImagesListParameters.Filters is of type IDictionary<string, IDictionary<string, bool>> because that's the best mapping to the REST API. Someone not familiar with the REST API will have trouble figuring out how to use that. And since the class is generated code, there's no XML documentation on Filters to help. In the proposed layer, the options class for ListAsync would have members such as bool? Dangling and string[]? NameFilter, with documentation.
3. The return value from ContainerOperations.InspectContainerAsync gives information about the container, but not nested data such as the image's name or details of the associated volumes. In the proposed layer, InspectAsync would return a rich object. The consumer would be able to use, for example, container.Image.Name or container.Volume.Where(volume => !volume.MountPoint.IsReadOnly). (This requires several REST calls, but I'm assuming that consumers generally aren't optimizing for speed. If they are, they can still use the lower layer for more control.)

Thoughts?

[galvesribeiro]

That is the goal I would like to achieve. To have a more user-friendly API surface.

However, IMHO, we don't have core in a modern way yet.

It is pretty useful, but again, IMHO, far from ideal.

I think we need to revamp the core first and drop the usage of the Go generator in favor of use Docker YAML spec and generate the core HTTP client based on it.

We have multiple enhancements on HttpClientFactory that we could use here as well, and also use a more performant serializer like System.Text.Json.

Once we have that core generating code more efficiently and serializing things as fast as possible, then we lay the ground to more high-level APIs on top of it.

On the generator side, I would use something to read that YAML (perhaps YamlDotNet) to build a model, and then use the brand new .Net 5 Source Generators to generate the code.

[dgvives]

@Emdot indeed, also share @galvesribeiro opinion

[Emdot]

Thanks for the responses!

Out of curiosity, why do you think the core revamp needs to happen before pursuing a higher-level API? If you're concerned about not having time to work on both, I'm happy to contribute substantially to the latter. It would still take some of your time, for PR review and the like, but I do solid work.

[galvesribeiro]

I think that the kind of features that you are looking for, would be significantly implemented differently based on the foundations exposed.

For example, there are primitives today for async enumeration and observer/observable patterns that would help us with the exposure of the streams (i.e. the container log stream) on the high-level APIs. Also the usage of Span<T> and Memory<T> would affect the way those high-level APIs are implemented since the Span doesn't work on async methods (due to its stack allocation nature).

This library has a potential to be used on huge projects just like the others on other languages but we need to have a sustainable way to keep up-to-date with Docker API changes. The way we do it Today with Go files just works (TM), but we would need to have too much work on the codegen in order to change it to generate the code that leverage the new facilities from .Net 5 and I would definitively feel more comfortable if we do that using "our own stuff" rather than hijacking the Go files.

I really plan to use this library for a bigger OSS project (a new container orchestrator based on Orleans - yeah, I'm tired of fighting Kubernetes issues), so I'm building myself more time to dedicate to Docker.DotNet first, because it will be the core, and then start/continue the new project.

Anyway, the APIs you suggested for sure, can implemented as extensions to the current ones without having to wait for anything bigger. But I would carefully consider on how much efforts will be put against a public API that will definitively change in a not that far future. If you feel comfortable with it, feel free to open dedicated issues for each API, lets discuss it, then PR -> merge. No problem.

Again, thanks for the contributions! Keep it coming! 😄

[Emdot]

Thanks for the thoughtful reply!

Yeah, I'll go ahead, then, with the understanding that some implementation may need to change based on core work.

Since you're talking about .NET 5 features, does that mean you're planning to drop support for older versions of .NET? Even just the difference between .NET Standard 2.0 and .NET Standard 2.1 would affect the code I write (such as with nullable annotations).

By the way, I think source generators probably won't help you. They generate .NET code from other .NET code, whereas you need to to generate C# from yaml. T4 templates are probably more applicable.

[galvesribeiro]

Source generators can generate code regardless of the input. I have it working with YAML in another project just fine.

Regarding the versioning, I'm not sure yet whether will go all the way net5.0 or something else. Still need to evaluate but, regardless, the language features are internal and dont need to be exposed to consumers. Nullable annotation is something to old versions only see if they enable it on the csproj, so not a big deal.

[dgvives]

@Emdot I don't see how #501 could improve or ease the use for this Docker.DotNet library.

Do you have some idea in mind?

[Emdot]

I do indeed! Keep in mind that this is just the foundation on which the more useful bits would be built, so it doesn't yet provide much benefit, but I can already give a few small examples.

Scenario 1

Goal: Connect to the daemon on the local machine and and check that the connection worked.

Before

var client = new DockerClientConfiguration().CreateClient();
await client.System.PingAsync();

After

var client = await Docker.StartAsync();

Difference: more discoverable, and the checks are built-in.

Scenario 2

Goal: Check whether the daemon supports API level 1.41 or higher.

Before:

var versionInfo = await client.System.GetVersionAsync();
var ok = new Version(versionInfo.APIInfo) >= new Version("1.41");

After:

var ok = client.ApiVersion >= new Version("1.41");

Difference: more discoverable, simpler

Scenario 3

Goal: Push and pull to private registries.

Before: The developer needs to pass credentials into every call. This may involve threading them through several function calls or registering them with dependency injection.

After: The developer registers the credentials once (with client.Registries.Add*). They're automatically used as needed for the lifetime of the process.

Difference: easier to manage complexity, more similar to the familiar docker login style

Next...

The real benefits come with later PRs. When we get to, say, client.Container.CreateAndStartAsync, I think you'll agree that this presents a huge simplification for consumers.

[Emdot]

Hi everyone!

My dev for the usability layer is going well, but it's severely blocked by lack of review. That isn't a criticism--I understand that open-source projects have limited resources--but at the same time I want to keep moving forward rapidly. I think the best solution is to make the usability layer a separate project for now. We can reintegrate at some point in the future. This also gives me more flexibility with .NET versions, release cadence, CI/CD pipeline, etc.

I've created the new project at Emdot/DockerSdk. It consumes this project's NuGet package, as before. Its documentation links to this project for "if you want more power". I'd appreciate if you'd link to that project for "if you want a friendly wrapper".

I hope we continue to work together well,
--Matt

[jterry75]

@galvesribeiro - Want to add this to the intro md page? It will also help @Emdot if we are able to update the release version so the library can depend on the fixes being made. I know you were working on that as well in the background

[galvesribeiro]

@Emdot do you think we can work that way for now? You can create the "SDK" library on your own outside this repo and reference this library. Then we add a remark on the README.md pointing to your package as @jterry75 suggested.

I think for now, this would help to keep the current codebase as minimal as possible while we move forward to the the new way of do things here. Once we have the new way publicly available here in another branch, we can work together to enhance it with your changes.

The idea is to not add extra API surface to the current API so we only keep it in "maintenance mode" while we work out the new stuff.

What do you think?

[Emdot]

@galvesribeiro Sounds good to me!