Unable to configure Harbor as container registry

[cgpoh]

Description:
I'm trying to configure SCDF to use our private Harbor registry in Kubernetes. I've created the .dockerconfigjson as image pull secret (harbor-registry) in the cluster and using the bitnami Helm chart, I added the following configuration:

containerRegistries:
  harbor-registry:
    authorization-type: dockeroauth2
    registry-host: registry.test
    user: myUser
    secret: mySecret
    extra:
      registryAuthUri: https://registry.test/service/token?service=harbor-registry&scope=repository:{repository}:pull

  extraVolumes:
    - name: harbor-registry
      secret:
        secretName: harbor-registry
  extraVolumeMounts:
    - name: harbor-registry
      readOnly: true
      mountPath: /etc/secrets/harbor-registry

deployer:
  imagePullSecrets:
    - harbor-registry

Release versions:

{
  "versions": {
    "implementation": {
      "name": "spring-cloud-dataflow-server",
      "version": "2.11.1"
    },
    "core": {
      "name": "Spring Cloud Data Flow Core",
      "version": "2.11.1"
    },
    "dashboard": {
      "name": "Spring Cloud Dataflow UI",
      "version": "3.4.1"
    },
    "shell": {
      "name": "Spring Cloud Data Flow Shell",
      "version": "2.11.1",
      "url": "https://repo.maven.apache.org/maven2/org/springframework/cloud/spring-cloud-dataflow-shell/2.11.1/spring-cloud-dataflow-shell-2.11.1.jar"
    }
  },
  "features": {
    "streams": true,
    "tasks": true,
    "schedules": true,
    "monitoringDashboardType": "NONE"
  },
  "runtimeEnvironment": {
    "appDeployer": {
      "deployerImplementationVersion": "2.11.1",
      "deployerName": "Spring Cloud Skipper Server",
      "deployerSpiVersion": "2.11.1",
      "javaVersion": "17.0.8",
      "platformApiVersion": "",
      "platformClientVersion": "",
      "platformHostVersion": "",
      "platformSpecificInfo": {
        "default": "kubernetes"
      },
      "platformType": "Skipper Managed",
      "springBootVersion": "2.7.16",
      "springVersion": "5.3.30"
    },
    "taskLaunchers": [
      {
        "deployerImplementationVersion": "unknown",
        "deployerName": "KubernetesTaskLauncher",
        "deployerSpiVersion": "unknown",
        "javaVersion": "17.0.8",
        "platformApiVersion": "v1",
        "platformClientVersion": "unknown",
        "platformHostVersion": "unknown",
        "platformSpecificInfo": {
          "namespace": "rep-test",
          "master-url": "https://172.16.0.1:443/"
        },
        "platformType": "Kubernetes",
        "springBootVersion": "2.7.16",
        "springVersion": "5.3.30"
      }
    ]
  },
  "monitoringDashboardInfo": {
    "url": "",
    "source": "default-scdf-source",
    "refreshInterval": 15
  },
  "security": {
    "isAuthentication": false,
    "isAuthenticated": false,
    "username": null,
    "roles": []
  }
}

Steps to reproduce:
I register my Springboot 3 app using the dashboard and have this in the docker uri: docker://registry.test/harbor-project-name/components/myapp:1.0.0

When clicking on my app in the dashboard to see more details, I get the following error in the SCDF server log:

2023-11-09 06:03:53.487  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies :_gorilla_csrf=MTY5OTUwOTgzM3xJa1kxTkZOeGFYZFpUVEEyZWxKclREUmFaalV5TXpWdVVWQnZaVzUzYVZZd05FNWxUa3B6VHpaTFIzTTlJZ289fJvBHXcFP3nsU5y0uhqxrAnAqbY9DtKbhy3XmnZKp583; Path=/; Expires=Thu, 09 Nov 2023 18:03:53 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:53 GMT
2023-11-09 06:03:53.735  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzM3xJbFIxV0RacE0xbzNkRVZYYWxKUFVtdzBiSFEwVlhWUVpVbE5Wa3d6VlcwMEsyOUdSR2cyZGxacFVGazlJZ289fHFyREhSNlEqYbNM59SQz73dtJo96NGBEUACvISlumwC; Path=/; Expires=Thu, 09 Nov 2023 18:03:53 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:53 GMT
2023-11-09 06:03:54.507  WARN 1 --- [nio-8080-exec-8] ApplicationConfigurationMetadataResolver : Failed to retrieve properties for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.BadRequest: 400 Authentication information is not given in the correct format. Check the value of Authorization header.: "?<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header.<EOL>RequestId:c0d84b19-f01e-008d-35d2-1257c4000000<EOL>Time:2023-11-09T06:03:54.4301786Z</Message></Error>"
2023-11-09 06:03:55.191  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNHxJbTFNUlhkYU1HOXpXV1JEZEZSVGRuSk5ZMEV2Y0dKNGIycFhRV2hYTkdObFF6RnFUVFJGVDFaM1FXODlJZ289fKNhTX3nhvvIE7w6ViGSCtQomqssQMB2MoaoyWaVbCYQ; Path=/; Expires=Thu, 09 Nov 2023 18:03:54 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:54 GMT
2023-11-09 06:03:55.476  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNXxJa0UxU1hSTVlrdzNkemwzUWs1cmVFeHhOMHAzTTAwNVZtRTJjMUJuWmxsUk9GbE5LM1JQYWtSWFYxRTlJZ289fFpCvie-lIYL2pBjfJOQ0i8qcryyHf2E0Pj5FtLHKMKg; Path=/; Expires=Thu, 09 Nov 2023 18:03:55 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:55 GMT
2023-11-09 06:03:55.725  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNXxJblZ4V2xwa1dVaDROemR4TVdrNGJWRTBjM2xVUm5kbVV6SjVLMVpYWmswd1dGWmpZVzVsUW10UFVYTTlJZ289fM7tb8ifi6vEka8jRg9A4yJGE5hueqJrgS6sbIoLg3EB; Path=/; Expires=Thu, 09 Nov 2023 18:03:55 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:55 GMT
2023-11-09 06:03:55.874  WARN 1 --- [nio-8080-exec-8] ApplicationConfigurationMetadataResolver : Failed to retrieve port names for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.BadRequest: 400 Authentication information is not given in the correct format. Check the value of Authorization header.: "?<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header.<EOL>RequestId:c0d84c7e-f01e-008d-71d2-1257c4000000<EOL>Time:2023-11-09T06:03:55.8013832Z</Message></Error>"
2023-11-09 06:03:56.353  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNXxJa2QxUzA5dEsxcFVNa2RHVTAwNE5WWnNXbnBaZUZCNGJHdG9aVXBIWkVOWGVXaGxZa2xLYTFsRVFWazlJZ289fNKey1Jy5AgH7jCMx41l7fgaJ6e_XoIdf-80Fwm-K5vm; Path=/; Expires=Thu, 09 Nov 2023 18:03:55 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:55 GMT
2023-11-09 06:03:56.652  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNnxJbW81Vm1SM1FrTlpSakJVYWpCRFRGQm5PRzFOVjNJMVYwcFlZV3hZTjBGNlIwNVlNalpRTlVGVFZ6ZzlJZ289fCcg6kCYfVwPNBDK86-lFyaUoqQ1fjv0o9NGmfJR5ddq; Path=/; Expires=Thu, 09 Nov 2023 18:03:56 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:56 GMT
2023-11-09 06:03:56.901  WARN 1 --- [nio-8080-exec-8] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTUwOTgzNnxJbXc1YTFaeWNWVmhTMWRqT1hvMlkwWTNibmt2YTNKWGJrOUlTMmRRTWxSb1dIbEtSbTlSWkZaWVlqQTlJZ289fJK2AiAjPSmp43I6wGdT0lK4lJm_oSz1xHKObER4eDij; Path=/; Expires=Thu, 09 Nov 2023 18:03:56 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Thu, 09 Nov 2023 18:03:56 GMT
2023-11-09 06:03:57.050  WARN 1 --- [nio-8080-exec-8] ApplicationConfigurationMetadataResolver : Failed to retrieve port names for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.BadRequest: 400 Authentication information is not given in the correct format. Check the value of Authorization header.: "?<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header.<EOL>RequestId:c0d84d98-f01e-008d-7dd2-1257c4000000<EOL>Time:2023-11-09T06:03:56.9777002Z</Message></Error>"

How can I resolve this issue?

[onobc]

Hi @cgpoh it looks like you may not be setting the imagePullSecrets for your app. Take a look through https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_private_docker_registry and see if that helps.

[cgpoh]

Hi @onobc, I already had imagepullsecrets configured. It's here

deployer:
imagePullSecrets:
- harbor-registry

[onobc]

I believe it should be one of

deployer.<application>.kubernetes.imagePullSecret: mysecret

or

deployer.kubernetes.imagePullSecret: mysecret

[cgpoh]

I'm using bitnami SCDF helm chart to deploy. I can't find deployer.kubernetes.imagePullSecret and from the documentation, to have global imagepullsecrets for application instead of 1 app per imagepullsecrets, I just need to configure

deployer:
imagePullSecrets:
- harbor-registry

Is it correct?

[onobc]

I see @cgpoh , yeh I think the bitnami helm chart just maps the install parameter deployer.imagePullSecrets -> deployer.kubernetes.imagePullSecrets so what you have should work for your case.

Are you setting this parameter on install, such as:

helm install my-release --set server.replicaCount=2 oci://REGISTRY_NAME/REPOSITORY_NAME/spring-cloud-dataflow

[cgpoh]

Nope, I cloned the helm chart project, change the values in the values file and helm install

[onobc]

Thanks @cgpoh. Your config does look as expected.

Looking at the log output you shared, there are 2 types of WARN messages in the output

• invalid cookie header (server trying to set cookies on the client)
• bad request (server barking that the auth header has invalid data)

Let's first assume the invalid cookie header is not the cause and that is an annoyance of the http client (used internally by resttemplate) not being able to parse standard cookie headers. This is a known issue in http client 4.

For the bad request, the Harbor server is complaining that the auth token is not in the expected format. Let's assume the issue is around this area.

I am sorry this is just not working for you as expected. We would look into it ourselves but we are busy getting the 2.11.2 release ready and would likely not be able to look into this for another couple of weeks. However, if you want to try a few things on your own, I would try this:

1. Make the credentials intentionally invalid in the secret and see if the error changes.

2. Debug through the Dataflow server around here org.springframework.cloud.dataflow.container.registry.ContainerRegistryService#getImageManifest and see what the headers look like.

3. Switch to basic auth for Harbor (temporarily to see if that works for you).

[cgpoh]

Thanks @onobc . I tried your suggestions:

1. Make the wrong credentials in the secret and I got a different error:

2023-11-10 07:04:50.301  WARN 1 --- [nio-8080-exec-9] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTU5OTg5MHxJak5JVURaM2RsY3laMWxTVkdKVFdubFZiR1pYUldSc1QySkZPRmxzUTNSc1dsTmhSQ3RsVFUxSGJYYzlJZ289fJ0swj4PVYFYV-1RuqAzQbC4SANrwhk7seRJrjvFwnu9; Path=/; Expires=Fri, 10 Nov 2023 19:04:50 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Fri, 10 Nov 2023 19:04:50 GMT
2023-11-10 07:04:50.309  WARN 1 --- [nio-8080-exec-9] ApplicationConfigurationMetadataResolver : Failed to retrieve properties for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.Unauthorized: 401 Unauthorized: "{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: harbor-project-name/components/myapp, action: pull: unauthorized to access repository: harbor-project-name/components/myapp, action: pull"}]}<EOL>"
2023-11-10 07:04:50.492  WARN 1 --- [nio-8080-exec-9] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTU5OTg5MHxJbnBIYWtkeU9WbFFXSEZ0ZVRabFUweFZWRk42SzJsQ05YVmFLMWhFZG5KVFZsRnpSbUZSV1VSTEt6QTlJZ289fNSqbwRK5Vlek9VIeCVigYbSH0o_PGOeXekBvI4kqpYW; Path=/; Expires=Fri, 10 Nov 2023 19:04:50 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Fri, 10 Nov 2023 19:04:50 GMT
2023-11-10 07:04:50.655  WARN 1 --- [nio-8080-exec-9] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTU5OTg5MHxJa1JQYmxCVldFTTJVMDl1ZDB4RlJVMXliMGRQTDIxTFNsaHRTMll3YmpWRVptbE9WVGRsUm5aVFZEZzlJZ289fFc-pi3iXI7RbsftmVk9JVE0vkhJAJG1m1lhSoMFOZfH; Path=/; Expires=Fri, 10 Nov 2023 19:04:50 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Fri, 10 Nov 2023 19:04:50 GMT
2023-11-10 07:04:50.655  WARN 1 --- [nio-8080-exec-9] ApplicationConfigurationMetadataResolver : Failed to retrieve port names for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.Unauthorized: 401 Unauthorized: "{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: harbor-project-name/components/myapp, action: pull: unauthorized to access repository: harbor-project-name/components/myapp, action: pull"}]}<EOL>"
2023-11-10 07:04:50.839  WARN 1 --- [nio-8080-exec-9] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTU5OTg5MHxJa1ZVUlVSTFJHRnRNMDVaTmtKU2FXVnVabmhwWTBsc1NEQTVSMUpoYVdZeEwzVnJRVFJpTHpWNWJVRTlJZ289fDa0qV2JduckZiftN0EgVP-bB0lgAQY4yWu60QtM7Shz; Path=/; Expires=Fri, 10 Nov 2023 19:04:50 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Fri, 10 Nov 2023 19:04:50 GMT
2023-11-10 07:04:51.000  WARN 1 --- [nio-8080-exec-9] o.a.h.c.protocol.ResponseProcessCookies  : Invalid cookie header: "Set-Cookie: _gorilla_csrf=MTY5OTU5OTg5MHxJbWQ0Ylc5SFMySjNPWHBFUjB0eE9HVkhlWEV3Y1RVeGRVVkVTRmN6UWxKb2NXbDBVSE5pTnl0Q1MyTTlJZ289fEKpC924FRmajxtsBmEF01DWNYyz75yUTKW2fcnULlo4; Path=/; Expires=Fri, 10 Nov 2023 19:04:50 GMT; Max-Age=43200; HttpOnly; Secure; SameSite=Strict". Invalid 'expires' attribute: Fri, 10 Nov 2023 19:04:50 GMT
2023-11-10 07:04:51.001  WARN 1 --- [nio-8080-exec-9] ApplicationConfigurationMetadataResolver : Failed to retrieve port names for resource Docker Resource [docker:registry.test/harbor-project-name/components/myapp:1.0.0] because of HttpClientErrorException.Unauthorized: 401 Unauthorized: "{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: harbor-project-name/components/myapp, action: pull: unauthorized to access repository: harbor-project-name/components/myapp, action: pull"}]}<EOL>"

2. Do you have a readme on how to start a dev environment for debugging?
3. Switch to basic auth and it doesn't work

[onobc]

Hi @cgpoh

3. Switch to basic auth and it doesn't work

What did you see in this case?

1. Make the wrong credentials in the secret and I got a different error:

So that is good that we see the change in response which tells us that at least it is finding the creds in the first place. Are you certain that the creds are valid and in the expected format for Harbor? Have you used them manually outside of Dataflow? I would try to issue some commands against Harbor (using curl against their REST API) and make sure the creds work for this case. Once we have success their we know it is in Dataflow somewhere that it is going wrong.

2. Do you have a readme on how to start a dev environment for debugging?

We do have some handy scripts for running a k8s cluster locally here. You can then just make sure that your pod is debug enabled w/ something like https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_enabling_debugging_with_patching.

[cgpoh]

Hi @onobc

Hi @cgpoh

3. Switch to basic auth and it doesn't work

What did you see in this case?

The same error as dockeroauth2

1. Make the wrong credentials in the secret and I got a different error:

So that is good that we see the change in response which tells us that at least it is finding the creds in the first place. Are you certain that the creds are valid and in the expected format for Harbor? Have you used them manually outside of Dataflow? I would try to issue some commands against Harbor (using curl against their REST API) and make sure the creds work for this case. Once we have success their we know it is in Dataflow somewhere that it is going wrong.

I tried curl command to get the token using the same credentials and using the returned token to pull the manifest without any problem.

2. Do you have a readme on how to start a dev environment for debugging?

We do have some handy scripts for running a k8s cluster locally here. You can then just make sure that your pod is debug enabled w/ something like https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_enabling_debugging_with_patching.

Thanks!

[cgpoh]

Hi @onobc , thanks for your remote debugging guidance. My issue is similar to this issue and this. I will get a 307 redirect code and need to remove the authentication headers to get the body from the blob. I will submit a PR to resolve this issue.