Question: pod not using the role defined on the ServiceAccount (IRSA) when fetching secrets (defaults to the EC2 node's role)

[Bhashit]

This is more of a question than an issue.

I am trying to integrate the kubernetes-external-secrets (KES) into EKS based kubernetes cluster. I was trying to use IRSA (IAM Roles For Service Accounts) to add authentication.

I have added the role-arn (that's attached to a policy that provides access to the secrets-manager) to the ServiceAccount for KES. I have verified that assuming this role using the following command allows the secrets to be fetched (I ran another container with AWS-CLI in it and had it attached to the same ServiceAccount I am using for KES):

aws sts assume-role-with-web-identity \
--role-arn $AWS_ROLE_ARN \
--role-session-name mh9test \
--web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
--duration-seconds 1000

This works correctly, and allows me to fetch secrets correctly (in that other container I ran).

However, the KES pod doesn't assume the correct role. It assumes the role that's attached to the EC2 instance on which it is running. Even though AWS_ROLE_ARN is set as an env-var.

My main question is: why am I required to put the roleArn within the ExternalSecret if I have already specified the role-arn as an annotation on the service-account (which does get added to the pod as AWS_ROLE_ARN env-var). Shouldn't it fallback to using the role that's specified for the ServiceAccount as a default?

It's certainly possible I'm misunderstanding things though. Also, if it turns out that what I was expecting should be the default behavior, I'd be happy to submit changes to the code.

[Flydiverny]

Currently there are some issues with IRSA usage due to #442 but this sounds like another issue.

Your assumption that the KES pods role should be used and there shouldn't be a need to specify a roleArn is correct, unless one of course wants to assume another role for that specific secret.

[Bhashit]

Looking at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/TokenFileWebIdentityCredentials.html, it seems like it could be either an AWS SDK version issue (I haven't checked the versions yet). Or a bug in the AWS SDK

it [sdk] will also read IAM role to be assumed from AWS_ROLE_ARN environment variable

EDIT

I ran the following commands in an interactive console on the KES pod (as outlined in here aws/aws-sdk-js#3090 (comment)):

const aws = require('aws-sdk');
const sts = new aws.STS({region: "us-east-2"});
sts.getCallerIdentity({}, function (error,data){console.log(data)});

And it assumes the correct identity.

[riccardomc]

I believe that in your test you are running your pod as root user, while the kubernetes-external-secrets pod is running as non-root by default. This is an important difference because in the "Pod Configuration" section of IRSA AWS Documentation you can read:

By default, only containers that run as root have the proper file system permissions to read the web identity token file. You can provide these permissions by having your containers run as root, or by providing the following security context for the containers in your manifest. The fsGroup ID is arbitrary, and you can choose any valid group ID.

Therefore you should try setting securityContext.fsGroup for your pod. The helm chart of this project already supports this.

As a side note... I believe this is a rather obscure and confusing mechanism. People in AWS seem to think the same, see this comment.

[Bhashit]

The pod's already running with securityContext.fsGroup set up to 65534

Also, the test pod's not running as the root user.

[tymokvo]

Also experiencing this with kubernetes-external-secrets-5.1.0 when applied with kubectl instead of the Helm chart.

The env of the Pod returns the expected role arn from the specified ServiceRole, but using the commands from this comment in a REPL in the external-secrets container returns the role of the EKS node on which the Pod is running. process.env.AWS_ROLE_ARN also returns the IAM role of the ServiceAccount. The securityGroup.fsGroup is already set to 65534.

[Flydiverny]

@tymokvo just to confirm, you are saying the REPL commands does not return your expected role (ie the service account role)?

[tymokvo]

Yes that's correct. It returns the role of the node.
(edit)
And, just to reiterate, I deployed this by exporting the helm chart from the repo into individual descriptors before following the directions for this in this repo's readme, if that matters.

[tbondarchuk]

had the same issue, fixed by adding to values:

securityContext:
  runAsNonRoot: true
  fsGroup: 65534

worth adding a note to readme perhaps to "2. IAM roles for service accounts."? comment in values.yaml is quite easy to miss especially for "still learning" k8s noobs like me :)

[ordonezf]

I'm having the same issue as @tymokvo (also installing with kubectl instead of helm instal), the service account has a role set and I can see it when I run env | grep AWS inside the pod, but if I run the REPL commands I get the role from the EKS node. Tried it on 4.2.0 and 5.1.0, same results.

Already tried adding the securityContext like @aliusmiles said but that didn't work.

This is part of the error KES throws
"message":"User: arn:aws:sts::ACCOUNT-ID:assumed-role/EKS-NODE-ROLE is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-east-1:ACCOUNT-ID:secret:MY-SECRET"

[Flydiverny]

Please provide more details on environment. EKS version at least :)

When running in the pod please verify if you can read the file set in the AWS env