Update buildkit to containerd leases #40122

tonistiigi · 2019-10-22T06:09:28Z

Updates buildkit to containerd leases based storage management moby/buildkit#1176 to support the future move to containerd storage in moby.

This update does not allow downgrades. On first daemon start, old storage is migrated to new.

This is only compatible with containerd 1.3+

daemon/images/images.go

distribution/xfer/download_test.go

layer/layer_store.go

builder/builder-next/adapters/containerimage/pull.go

dmcgowan · 2019-10-22T23:35:29Z

builder/builder-next/adapters/containerimage/pull.go

+		}
+		defer parent.Release(context.TODO())
+	}
+	return p.is.CacheAccessor.GetByBlob(context.TODO(), ocispec.Descriptor{


Why is this also a context.TODO?

Signed-off-by: Tonis Tiigi <[email protected]>

AkihiroSuda · 2019-11-18T18:05:09Z

@dmcgowan PTAL

thaJeztah

noticed two tiny issues; no need to fix those if no other changes are requested (I just noticed them now, but see there's some other locations already having the same, so I can do a follow-up to change them all in one go)

thaJeztah · 2019-11-22T15:18:31Z

builder/builder-next/adapters/snapshot/snapshot.go

 	reg, ok := opt.LayerStore.(graphIDRegistrar)
 	if !ok {
-		return nil, errors.Errorf("layerstore doesn't support graphID registration")
+		return nil, nil, errors.Errorf("layerstore doesn't support graphID registration")


Surprised that no linter complains about this; looks like this should be errors.New (because no formatting is used)

Suggested change

return nil, nil, errors.Errorf("layerstore doesn't support graphID registration")

return nil, nil, errors.New("layerstore doesn't support graphID registration")

thaJeztah · 2019-11-22T15:19:00Z

builder/builder-next/adapters/snapshot/snapshot.go

 }

 func (s *snapshotter) Remove(ctx context.Context, key string) error {
+	return errors.Errorf("calling snapshot.remove is forbidden")


Suggested change

return errors.Errorf("calling snapshot.remove is forbidden")

return errors.New("calling snapshot.remove is forbidden")

dmcgowan

LGTM

thaJeztah

thanks @dmcgowan

I'll copy your LGTM and merge

full diff: moby/moby@a09e6e3...a9507c6 Includes: - moby/moby#40077 Update "auto-generate" comments to improve detection by linters - moby/moby#40143 registry: add a critical section to protect authTransport.modReq - moby/moby#40212 Move DefaultCapabilities() to caps package - moby/moby#40021 Use newer x/sys/windows SecurityAttributes struct (carry 40017) - carries moby/moby#40017 Use newer x/sys/windows SecurityAttributes struct - moby/moby#40135 pkg/system: make OSVersion an alias for hcsshim OSVersion - follow-up to moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion() - moby/moby#40250 Bump hcsshim to b3f49c06ffaeef24d09c6c08ec8ec8425a0303e2 - moby/moby#40243 Use certs.d from XDG_CONFIG_HOME when in rootless mode - fixes moby/moby#40236 Docker rootless dies when unable to read /etc/docker/certs.d - moby/moby#40283 Fix possible runtime panic in Lgetxattr - moby/moby#40178 builder/remotecontext: small refactor - moby/moby#40179 builder/remotecontext: allow ssh:// for remote context URLs - fixes docker#2164 Docker build cannot resolve git context with html escapes - moby/moby#40302 client.ImagePush(): default to ":latest" instead of "all tags" - relates to docker#2214 [proposal] change "docker push" behavior to default to ":latest" instead of "all tags" - relates to docker#2220 implement docker push `-a`/ `--all-tags` - moby/moby#40263 Normalize comment formatting - moby/moby#40238 Allow client consumers like traefik to compile on illumos - moby/moby#40108 bump google.golang.org/grpc v1.23.1 - moby/moby#40312 update vendor golang.org/x/sys to 6d18c012aee9febd81bbf9806760c8c4480e870d - moby/moby#40247 pkg/system: deprecate constants in favor of golang.org/x/sys/windows - moby/moby#40246 pkg/system: minor cleanups and remove use of deprecated system.GetOSVersion() - moby/moby#40122 Update buildkit to containerd leases - vendor: update buildkit to leases support (4f4e03067523b2fc5ca2f17514a5e75ad63e02fb) - vendor: update containerd to acdcf13d5eaf0dfe0eaeabe7194a82535549bc2b - vendor: update runc to d736ef14f0288d6993a1845745d6756cfc9ddd5a (v1.0.0-rc9) Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/moby@a09e6e3...a9507c6 Includes: - moby/moby#40077 Update "auto-generate" comments to improve detection by linters - moby/moby#40143 registry: add a critical section to protect authTransport.modReq - moby/moby#40212 Move DefaultCapabilities() to caps package - moby/moby#40021 Use newer x/sys/windows SecurityAttributes struct (carry 40017) - carries moby/moby#40017 Use newer x/sys/windows SecurityAttributes struct - moby/moby#40135 pkg/system: make OSVersion an alias for hcsshim OSVersion - follow-up to moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion() - moby/moby#40250 Bump hcsshim to b3f49c06ffaeef24d09c6c08ec8ec8425a0303e2 - moby/moby#40243 Use certs.d from XDG_CONFIG_HOME when in rootless mode - fixes moby/moby#40236 Docker rootless dies when unable to read /etc/docker/certs.d - moby/moby#40283 Fix possible runtime panic in Lgetxattr - moby/moby#40178 builder/remotecontext: small refactor - moby/moby#40179 builder/remotecontext: allow ssh:// for remote context URLs - fixes docker/cli#2164 Docker build cannot resolve git context with html escapes - moby/moby#40302 client.ImagePush(): default to ":latest" instead of "all tags" - relates to docker/cli#2214 [proposal] change "docker push" behavior to default to ":latest" instead of "all tags" - relates to docker/cli#2220 implement docker push `-a`/ `--all-tags` - moby/moby#40263 Normalize comment formatting - moby/moby#40238 Allow client consumers like traefik to compile on illumos - moby/moby#40108 bump google.golang.org/grpc v1.23.1 - moby/moby#40312 update vendor golang.org/x/sys to 6d18c012aee9febd81bbf9806760c8c4480e870d - moby/moby#40247 pkg/system: deprecate constants in favor of golang.org/x/sys/windows - moby/moby#40246 pkg/system: minor cleanups and remove use of deprecated system.GetOSVersion() - moby/moby#40122 Update buildkit to containerd leases - vendor: update buildkit to leases support (4f4e03067523b2fc5ca2f17514a5e75ad63e02fb) - vendor: update containerd to acdcf13d5eaf0dfe0eaeabe7194a82535549bc2b - vendor: update runc to d736ef14f0288d6993a1845745d6756cfc9ddd5a (v1.0.0-rc9) Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 627a4cf7ccd0b7e92c6798c73de4dd4efc43175c Component: cli

full diff: moby/moby@a09e6e3...a9507c6 Includes: - moby/moby#40077 Update "auto-generate" comments to improve detection by linters - moby/moby#40143 registry: add a critical section to protect authTransport.modReq - moby/moby#40212 Move DefaultCapabilities() to caps package - moby/moby#40021 Use newer x/sys/windows SecurityAttributes struct (carry 40017) - carries moby/moby#40017 Use newer x/sys/windows SecurityAttributes struct - moby/moby#40135 pkg/system: make OSVersion an alias for hcsshim OSVersion - follow-up to moby/moby#39100 Use Microsoft/hcsshim constants and deprecate pkg/system.GetOsVersion() - moby/moby#40250 Bump hcsshim to b3f49c06ffaeef24d09c6c08ec8ec8425a0303e2 - moby/moby#40243 Use certs.d from XDG_CONFIG_HOME when in rootless mode - fixes moby/moby#40236 Docker rootless dies when unable to read /etc/docker/certs.d - moby/moby#40283 Fix possible runtime panic in Lgetxattr - moby/moby#40178 builder/remotecontext: small refactor - moby/moby#40179 builder/remotecontext: allow ssh:// for remote context URLs - fixes docker#2164 Docker build cannot resolve git context with html escapes - moby/moby#40302 client.ImagePush(): default to ":latest" instead of "all tags" - relates to docker#2214 [proposal] change "docker push" behavior to default to ":latest" instead of "all tags" - relates to docker#2220 implement docker push `-a`/ `--all-tags` - moby/moby#40263 Normalize comment formatting - moby/moby#40238 Allow client consumers like traefik to compile on illumos - moby/moby#40108 bump google.golang.org/grpc v1.23.1 - moby/moby#40312 update vendor golang.org/x/sys to 6d18c012aee9febd81bbf9806760c8c4480e870d - moby/moby#40247 pkg/system: deprecate constants in favor of golang.org/x/sys/windows - moby/moby#40246 pkg/system: minor cleanups and remove use of deprecated system.GetOSVersion() - moby/moby#40122 Update buildkit to containerd leases - vendor: update buildkit to leases support (4f4e03067523b2fc5ca2f17514a5e75ad63e02fb) - vendor: update containerd to acdcf13d5eaf0dfe0eaeabe7194a82535549bc2b - vendor: update runc to d736ef14f0288d6993a1845745d6756cfc9ddd5a (v1.0.0-rc9) Signed-off-by: Sebastiaan van Stijn <[email protected]>

AkihiroSuda approved these changes Oct 22, 2019

View reviewed changes

tonistiigi force-pushed the buildkit-leases2 branch from 9668f83 to 57cce67 Compare October 22, 2019 06:16

thaJeztah added area/builder Build area/runtime Runtime status/2-code-review labels Oct 22, 2019

thaJeztah reviewed Oct 22, 2019

View reviewed changes

daemon/images/images.go Outdated Show resolved Hide resolved

thaJeztah reviewed Oct 22, 2019

View reviewed changes

distribution/xfer/download_test.go Outdated Show resolved Hide resolved

thaJeztah reviewed Oct 22, 2019

View reviewed changes

layer/layer_store.go Outdated Show resolved Hide resolved

thaJeztah reviewed Oct 22, 2019

View reviewed changes

builder/builder-next/adapters/containerimage/pull.go Outdated Show resolved Hide resolved

tonistiigi force-pushed the buildkit-leases2 branch 2 times, most recently from b7debc9 to 0aeb851 Compare October 22, 2019 20:39

dmcgowan reviewed Oct 22, 2019

View reviewed changes

tonistiigi mentioned this pull request Oct 25, 2019

Bump buildkit v0.6.2 #40136

Closed

tonistiigi force-pushed the buildkit-leases2 branch 2 times, most recently from 23b54f7 to 88f5526 Compare October 29, 2019 17:57

tonistiigi added 5 commits November 5, 2019 10:14

vendor: update buildkit to leases support

fb1601d

Signed-off-by: Tonis Tiigi <[email protected]>

builder-next: update adapters to new buildkit interfaces

fe16d95

Signed-off-by: Tonis Tiigi <[email protected]>

builder-next: track layers and graphdrivers with leases

f14c9d4

Signed-off-by: Tonis Tiigi <[email protected]>

vendor: update containerd to acdcf13d5eaf0dfe0eaeabe7194a82535549bc2b

f632e2d

Signed-off-by: Tonis Tiigi <[email protected]>

builder-next: clear temp leases on startup

21dfcc7

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi force-pushed the buildkit-leases2 branch from 88f5526 to 21dfcc7 Compare November 5, 2019 20:50

thaJeztah reviewed Nov 22, 2019

View reviewed changes

dmcgowan approved these changes Nov 26, 2019

View reviewed changes

thaJeztah approved these changes Nov 26, 2019

View reviewed changes

thaJeztah merged commit 4afda3b into moby:master Nov 26, 2019

thaJeztah mentioned this pull request Jan 7, 2020

vendor: bump docker and dependencies a9507c6f76627fdc092edc542d5a7ef4a6df5eec docker/cli#2242

Merged

thaJeztah mentioned this pull request Feb 4, 2020

[19.03] update containerd 1.12.12, runc v1.0.0-rc10 #40453

Merged

thaJeztah added this to the 20.03.0 milestone Apr 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update buildkit to containerd leases #40122

Update buildkit to containerd leases #40122

Uh oh!

tonistiigi commented Oct 22, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmcgowan Oct 22, 2019

Uh oh!

AkihiroSuda commented Nov 18, 2019

Uh oh!

thaJeztah left a comment

Uh oh!

thaJeztah Nov 22, 2019

Uh oh!

thaJeztah Nov 22, 2019

Uh oh!

dmcgowan left a comment

Uh oh!

thaJeztah left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	return nil, nil, errors.Errorf("layerstore doesn't support graphID registration")
	return nil, nil, errors.New("layerstore doesn't support graphID registration")

	return errors.Errorf("calling snapshot.remove is forbidden")
	return errors.New("calling snapshot.remove is forbidden")

Update buildkit to containerd leases #40122

Update buildkit to containerd leases #40122

Uh oh!

Conversation

tonistiigi commented Oct 22, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmcgowan Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Nov 18, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah Nov 22, 2019

Choose a reason for hiding this comment

Uh oh!

thaJeztah Nov 22, 2019

Choose a reason for hiding this comment

Uh oh!

dmcgowan left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants