s3 storage on scaleway: configurable default region instead 'us-east-1'

[sdelrio]

• Stash: v0.9.0-rc.5
• kubernetes 1.16.6
• Object storage provider: Scaleway (only have two regions: fr-par & nl-ams, tested with both)

When trying to use Scaleway object storage it gives an error because is not recognizing us-east-1 , there is a posibility to configure the default region if none was not found? For example, something like this:

  backend:
    s3:
      endpoint: 's3.nl-ams.scw.cloud'
      bucket: 'stash-bucket'
      default_region: 'nl-ams'    <-----
      prefix: '/source/sample-statefulset'

• collected logs from stash (kubectl logs --tail=10 -f stash-demo-2 -n demo -c stash)

Fatal: unable to open config file: Stat: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'
Is there a repository at the following location?
s3:s3.nl-ams.scw.cloud/stash-bucket/source/sample-statefulset
[golang-sh]$ /bin/restic_0.9.5 init --cache-dir /tmp/restic-cache
Fatal: create repository at s3:s3.nl-ams.scw.cloud/stash-bucket/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'

W0211 16:14:06.951235       1 backupsession.go:478] Failed to take backup for BackupSession s3-backup-1581437645. Reason: Fatal: create repository at s3:s3.nl-ams.scw.cloud/stash-bucket/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'
I0211 16:14:06.959312       1 status.go:104] Updating status of BackupSession: demo/s3-backup-1581437645 for host: host-2
W0211 16:14:07.000329       1 backupsession.go:405] failed to complete backup. Reason: Fatal: create repository at s3:s3.nl-ams.scw.cloud/stash-bucket/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'

• repository.yaml used to get his log

apiVersion: stash.appscode.com/v1alpha1
kind: Repository
metadata:
  name: s3-scw-repo
  namespace: demo
spec:
  backend:
    s3:
      endpoint: 's3.nl-ams.scw.cloud'
      bucket: 'stash-bucket'
      prefix: '/source/sample-statefulset'
    storageSecretName: stash-secret

[sebastien-prudhomme]

Hi, same problem for me. It seems there is an option region in the source code:

https://github.com/stashed/stash/blob/master/api/crds/stash.appscode.com_repositories.yaml#L1294

But it's not working

[sdelrio]

I also tried that region part but it seems that it doesn't find region when using scaleway, so it goes to default (us-east-1) and then fails

[hossainemruz]

@sdelrio Our underlying API already support providing region. We just need to handle in Stash side. I think we can add this in next release.

Ref: https://github.com/kmodules/objectstore-api/blob/5881939b57f0dfe62709a2f9ea644adbd50e105e/api/v1/types.go#L100

[hossainemruz]

@sdelrio and @sebastien-prudhomme this issue should be fixed by this PR: #1050. This fix should be available in next release.

In the meantime, you try a custom image I have build from the fix. Follow the following steps:

1. Uninstall existing Stash operator.
2. Install new operator with the fix.

   $ helm repo add appscode https://charts.appscode.com/stable/
   $ helm repo update
   
   $ helm install stash-operator appscode/stash \
     --version v0.9.0-rc.4 \
     --namespace kube-system \
     --set operator.registry=appscodeci \
     --set operator.tag=aws-region_linux_amd64

3. Then, create repository with region specified.

kind: Repository
metadata:
  name: s3-scw-repo
  namespace: demo
spec:
  backend:
    s3:
      endpoint: 's3.nl-ams.scw.cloud'
      bucket: 'stash-bucket'
      region: 'nl-ams'
      prefix: '/source/sample-statefulset'
    storageSecretName: stash-secret

Let us know whether it worked or not.

[sebastien-prudhomme]

@hossainemruz the problem is still there, here is the logs of the stash sidecar:

I0213 22:45:06.612805       1 commands.go:82] Ensuring restic repository in the backend
[golang-sh]$ /bin/restic_0.9.5 snapshots --json --cache-dir /tmp/restic-cache
Fatal: unable to open config file: Stat: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'
Is there a repository at the following location?
s3:s3.fr-par.scw.cloud/********/stash/mongodb
[golang-sh]$ /bin/restic_0.9.5 init --cache-dir /tmp/restic-cache
Fatal: create repository at s3:s3.fr-par.scw.cloud/********/stash/mongodb failed: client.BucketExists: The authorization header is malform
ed; the region 'us-east-1' is wrong; expecting 'fr-par'

[sebastien-prudhomme]

@hossainemruz i've also checked that the stash sidecar is using the new image that you supplied

[sebastien-prudhomme]

@hossainemruz also tested the new mongodb image, and the fixed chart with the right --region option. Same problem.

Seems that Stash is using Restic 0.9.5. Perhaps the new version 0.9.6 is needed? See here: restic/restic#2350

[sdelrio]

Still getting zone error:

[golang-sh]$ /bin/restic_0.9.5 init --cache-dir /tmp/restic-cache
Fatal: create repository at s3:s3.nl-ams.scw.cloud/nuc-stash/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'

W0214 00:48:03.401682       1 backupsession.go:478] Failed to take backup for BackupSession s3-backup-1581641282. Reason: Fatal: create repository at s3:s3.nl-ams.scw.cloud/nuc-stash/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'
I0214 00:48:03.406774       1 status.go:104] Updating status of BackupSession: demo/s3-backup-1581641282 for host: host-0
W0214 00:48:03.428335       1 backupsession.go:405] failed to complete backup. Reason: Fatal: create repository at s3:s3.nl-ams.scw.cloud/nuc-stash/source/sample-statefulset failed: client.BucketExists: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'nl-ams'

Some info from the pods to confirm new image running:

 $ kubectl describe pod stash-operator-769b465594-d2nwn -n kube-system|grep Image
    Image:         appscodeci/stash:aws-region_linux_amd64
    Image ID:      docker-pullable://appscodeci/stash@sha256:a7b4d242e83530bdd4c55c51999fde2a102c1fc94c25d55cebdd9139a2fe747d
    Image:         prom/pushgateway:v0.5.2
    Image ID:      docker-pullable://prom/pushgateway@sha256:4c9fa582c167f6fec89c6ebbd454ec1b5dba3ee0e1e21118fac6737c06861f92

 $ kubectl  -n demo describe pod stash-demo-0 |grep Image
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Image:         appscodeci/stash:aws-region_linux_amd64
    Image ID:      docker-pullable://appscodeci/stash@sha256:a7b4d242e83530bdd4c55c51999fde2a102c1fc94c25d55cebdd9139a2fe747d

$ cat demo-repository.yaml
apiVersion: stash.appscode.com/v1alpha1
kind: Repository
metadata:
  name: s3-scw-repo
  namespace: demo
spec:
  backend:
    s3:
      endpoint: 's3.nl-ams.scw.cloud'
      region: 'nl-ams'
      bucket: 'stash-bucket'
      prefix: '/source/sample-statefulset'
    storageSecretName: stash-secret

[hossainemruz]

@sebastien-prudhomme you are right. I didn't noticed that region support was added in restic 0.9.6. I have updated the PR and images. Use v0.9.0-rc.5-4-gacb1e6e3_linux_amd64 tag for Stash operator image.

For, MongoDB you have to use appscodeci/stash-mongodb:7c95ca6_linux_amd64 image for backup and restore Function.

[sebastien-prudhomme]

@hossainemruz For MongoDB, i used the right version: appscodeci/stash-mongodb:478a9cc_linux_amd64

S3 access is now ok with the MongoDB function. Still have issue but not related to S3, i guess i've made a mistake in auth/password configuration.

[sebastien-prudhomme]

@hossainemruz it's also ok for deployment volume backup, here is the status of a test backupsession:

Status:
  Phase:             Succeeded
  Session Duration:  25.102267635s
  Targets:
    Phase:  Succeeded
    Ref:
      Kind:  Deployment
      Name:  mongodb
    Stats:
      Duration:  24.91332426s
      Hostname:  host-0
      Phase:     Succeeded
      Snapshots:
        File Stats:
          Modified Files:    0
          New Files:         401
          Total Files:       401
          Unmodified Files:  0
        Name:                ff6500c8
        Path:                /bin
        Processing Time:     0:03
        Total Size:          457.230 MiB
        Uploaded:            28.309 MiB
    Total Hosts:             1

Thanks for your work.