bug(aws): Trivy can't connect to `IMDS` to get credentials

[DmitriyLewen]

Description

We started using xhttp.Client for aws config in #9322 to add the Insecure option.

But with this client, the aws-sdk cannot connect to IMDS to get credentials.

Solution

Use the aws BuildableClient with the Insecure option.

Discussed in #9429

[knqyf263]

Did you replicate this issue? I did the following, but it works.

Environment

• OS: Amazon Linux 2023 (ami-04931b7f6d6cd9d61)
• Instance Type: t3.small
• Region: eu-north-1
• Authentication: EC2 Instance Profile with ECR read permissions via IMDSv2

Test Setup Commands

1. Create IAM Role and Policy

# Create IAM role for ECR access
aws iam create-role --role-name TrivyECRScanRole --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}'

# Create custom ECR read policy
aws iam create-policy --policy-name TrivyECRReadOnlyPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability", 
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage"
    ],
    "Resource": "*"
  }]
}'

# Attach policy and create instance profile
aws iam attach-role-policy --role-name TrivyECRScanRole --policy-arn arn:aws:iam::ACCOUNT_ID:policy/TrivyECRReadOnlyPolicy
aws iam create-instance-profile --instance-profile-name TrivyECRScanInstanceProfile
aws iam add-role-to-instance-profile --instance-profile-name TrivyECRScanInstanceProfile --role-name TrivyECRScanRole

2. Launch EC2 Instance

# Launch EC2 with IAM role
aws ec2 run-instances \
  --image-id ami-04931b7f6d6cd9d61 \
  --instance-type t3.small \
  --key-name aws_id_ed25519 \
  --security-group-ids sg-SECURITY_GROUP_ID \
  --iam-instance-profile Name=TrivyECRScanInstanceProfile

IMDSv2 Authentication Verification

# Test IMDSv2 token acquisition
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" 2>/dev/null)
echo "Token acquired: ${TOKEN:0:20}..."

Result:

Token acquired: AQAEAOZO1xe2nFVSVVfq...

# Verify instance role
curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null

Result:

TrivyECRScanRole

# Check credentials
curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/TrivyECRScanRole 2>/dev/null | head -5

Result:

{
  "Code" : "Success",
  "LastUpdated" : "2025-09-04T18:49:53Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIA...",

Trivy Installation and Testing

Binary Version Test

# Install Trivy binary
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sudo sh -s -- -b /usr/local/bin
trivy --version

Result:

Version: 0.66.0

# Test 1: Binary version ECR scan
trivy image ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20

Result:

Report Summary

┌──────────────────────────────────────────────────────────────────────────────┬────────┬─────────────────┬─────────┐
│                                    Target                                    │  Type  │ Vulnerabilities │ Secrets │
├──────────────────────────────────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20 (alpine   │ alpine │        0        │    -    │
│ 3.20.7)                                                                      │        │                 │         │
└──────────────────────────────────────────────────────────────────────────────┴────────┴─────────────────┴─────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

2025-09-04T18:53:04Z	INFO	[vulndb] Need to update DB
2025-09-04T18:53:04Z	INFO	[vulndb] Downloading vulnerability DB...
2025-09-04T18:53:04Z	INFO	[vulndb] Downloading artifact...	repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-09-04T18:53:09Z	INFO	[vulndb] Artifact successfully downloaded	repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-09-04T18:53:09Z	INFO	[vuln] Vulnerability scanning is enabled
2025-09-04T18:53:09Z	INFO	[secret] Secret scanning is enabled
2025-09-04T18:53:09Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-09-04T18:53:09Z	INFO	[secret] Please see https://trivy.dev/v0.66/docs/scanner/secret#recommendation for faster secret detection
2025-09-04T18:53:09Z	INFO	Detected OS	family="alpine" version="3.20.7"
2025-09-04T18:53:09Z	INFO	[alpine] Detecting vulnerabilities...	os_version="3.20" repository="3.20" pkg_num=14
2025-09-04T18:53:09Z	INFO	Number of language-specific files	num=0

Container Version Test

# Install Docker
sudo dnf install -y docker
sudo systemctl start docker
sudo usermod -a -G docker ec2-user

# Test 2: Container version ECR scan
docker run --rm aquasec/trivy:0.66.0 image ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20

Result:

Report Summary

┌──────────────────────────────────────────────────────────────────────────────┬────────┬─────────────────┬─────────┐
│                                    Target                                    │  Type  │ Vulnerabilities │ Secrets │
├──────────────────────────────────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20 (alpine  │ alpine │        0        │    -    │
│ 3.20.7)                                                                      │        │                 │         │
└──────────────────────────────────────────────────────────────────────────────┴────────┴─────────────────┴─────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

Unable to find image 'aquasec/trivy:0.66.0' locally
0.66.0: Pulling from aquasec/trivy
9824c27679d3: Pull complete
6eae7a225cc2: Pull complete
c7cde9aa180e: Pull complete
e00e2cad636b: Pull complete
Digest: sha256:086971aaf400beebd94e8300fd8ea623774419597169156cec56eec5b00dfb1e
Status: Downloaded newer image for aquasec/trivy:0.66.0

2025-09-04T19:04:39Z	INFO	[vulndb] Need to update DB
2025-09-04T19:04:39Z	INFO	[vulndb] Downloading vulnerability DB...
2025-09-04T19:04:39Z	INFO	[vulndb] Downloading artifact...	repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-09-04T19:04:45Z	INFO	[vulndb] Artifact successfully downloaded	repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-09-04T19:04:45Z	INFO	[vuln] Vulnerability scanning is enabled
2025-09-04T19:04:45Z	INFO	[secret] Secret scanning is enabled
2025-09-04T19:04:45Z	INFO	[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-09-04T19:04:45Z	INFO	[secret] Please see https://trivy.dev/v0.66/docs/scanner/secret#recommendation for faster secret detection
2025-09-04T19:04:45Z	INFO	Detected OS	family="alpine" version="3.20.7"
2025-09-04T19:04:45Z	INFO	[alpine] Detecting vulnerabilities...	os_version="3.20" repository="3.20" pkg_num=14
2025-09-04T19:04:45Z	INFO	Number of language-specific files	num=0

Summary

✅ Test Results

• Binary version (v0.66.0): ✅ Successfully scanned ECR private image
• Container version (v0.66.0): ✅ Successfully scanned ECR private image
• IMDSv2 Authentication: ✅ Working correctly with EC2 instance role
• ECR Authentication: ✅ Automatic credential acquisition from IMDSv2

Expected vs Actual

• Expected: v0.66.0 should fail to scan ECR images due to reported bug
• Actual: Both binary and container versions successfully scanned ECR private images without errors

[chriskellyrelay]

@knqyf263 I experienced this problem as trivy not being able to connect to our ECR Pull through cache repositories for the db and java-db images, and the open fix for this bug fixes that problem for me.

[knqyf263]

In my environment, it also works with ECR Pull through cache.

Pull-Through Cache Configuration

GitHub Container Registry (GHCR) Pull-Through Cache

Setup Commands

# Create secret in AWS Secrets Manager
aws secretsmanager create-secret \
  --name ecr-pullthroughcache/ghcr \
  --description "GHCR credentials for ECR pull-through cache" \
  --secret-string '{"username":"USERNAME","accessToken":"gho_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}' \
  --region eu-north-1

# Create GHCR pull-through cache rule
aws ecr create-pull-through-cache-rule \
  --ecr-repository-prefix ghcr \
  --upstream-registry-url ghcr.io \
  --credential-arn arn:aws:secretsmanager:eu-north-1:ACCOUNT_ID:secret:ecr-pullthroughcache/ghcr-XXXXXX \
  --region eu-north-1

Test Commands and Results

Basic Pull-Through Cache Testing:

# Pull Trivy image via GHCR pull-through cache
docker pull ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy:latest
# Result: ✅ Successfully pulled

# Scan GHCR pull-through cached Trivy image
trivy image ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy:latest
# Result: ✅ Alpine 3.22.1, 2 vulnerabilities (1 MEDIUM, 1 HIGH)

DB Repository via Pull-Through Cache:

# Using GHCR pull-through cache for DB repository
trivy image \
  --db-repository ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy-db:2 \
  ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20
# Result: ✅ Successfully downloaded DB (69.94 MiB), scan completed

Results

✅ All tests passed successfully

[knqyf263]

I have reviewed the logs from all reporters, and it seems that everyone is running Trivy on GitLab. Since I cannot reproduce the issue either on my local environment or on EC2, it might be a problem that only occurs on GitLab.

• Fetching database through ECR seems to fail starting with release 0.66.0 #9430
• Version 0.66.0 not authorized with AWS #9429

[chriskellyrelay]

Interesting! Yes, we are running this is a Gitlab self-hosted job on a self-hosted runner in AWS.

[knqyf263]

It still works for me...

# Simple GitLab CI job for testing Trivy with ECR
stages:
  - test
  - scan

variables:
  AWS_REGION: "eu-north-1"
  ACCOUNT_ID: "XXXXXXXXX"


# Test basic functionality
test_setup:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache aws-cli
  script:
    - echo "Testing AWS IMDS authentication..."
    - aws sts get-caller-identity --region $AWS_REGION
    - echo "Setup complete"
  tags:
    - trivy
    - ecr
    - aws

# Trivy v0.66.0 ECR scan
trivy_066_scan:
  stage: scan
  image:
    name: aquasec/trivy:0.66.0
    entrypoint: [""]
  script:
    - echo "Running Trivy v0.66.0 ECR image scan..."
    - trivy image $ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20
        --db-repository $ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy-db:2
        --image-src remote
  tags:
    - trivy
    - ecr
    - aws

# Trivy v0.65.0 ECR scan
trivy_065_scan:
  stage: scan
  image:
    name: aquasec/trivy:0.65.0
    entrypoint: [""]
  script:
    - echo "Running Trivy v0.65.0 ECR image scan for comparison..."
    - trivy image $ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20
        --db-repository $ACCOUNT_ID.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy-db:2
        --image-src remote
  tags:
    - trivy
    - ecr
    - aws

INFO	[vulndb] Artifact successfully downloaded	repo="XXXXXXXXXXXX.dkr.ecr.eu-north-1.amazonaws.com/ghcr/aquasecurity/trivy-db:2"

┌──────────────────────────────────────────────────────────────────────────────┬────────┬─────────────────┬─────────┐
│                                    Target                                    │  Type  │ Vulnerabilities │ Secrets │
├──────────────────────────────────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ XXXXXXXXXXXX.dkr.ecr.eu-north-1.amazonaws.com/trivy-test:alpine-3.20 (alpine │ alpine │        0        │    -    │
│ 3.20.7)                                                                      │        │                 │         │
└──────────────────────────────────────────────────────────────────────────────┴────────┴─────────────────┴─────────┘

[eldondevat]

Are you able to try it without IMDSv2 enabled?