MinIO SDK ignores SSL configuration and forces HTTP connections

[hamzarhaiem]

Description

Testkube API Server 2.9.3 systematically connects to S3/Ceph storage using HTTP on port 80, completely ignoring all SSL/HTTPS configuration parameters. The MinIO Go SDK embedded in Testkube appears to reject HTTPS configuration regardless of the environment variables or endpoint format used.

Environment

• Testkube API Server Version: 2.9.3 (commit: ad2afe0b1f72a2d79ac2b2e378ee962d53085e37)
• Helm Chart Version: 2.9.5
• Kubernetes Version: RKE2 (Rancher)
• Storage Backend: Ceph S3 (HTTPS only on port 443)
• Deployment Mode: Standalone (OSS)

Note: This is the first Testkube version with PostgreSQL support (required for our deployment), which is why we cannot downgrade to older versions.

Expected Behavior

When STORAGE_SSL=true is configured, the MinIO client should connect to the S3 endpoint using HTTPS on port 443.

Actual Behavior

The MinIO SDK always connects using HTTP on port 80, regardless of configuration:

{
"level": "warn",
"msg": "Failed to check if the bucket exists; will retry",
"bucket": "my-bucket",
"error": "Get "http://s3.example.com/my-bucket/?location=\": dial tcp 10.x.x.x:80: i/o timeout"
}

Configuration Attempts (All Failed)

Attempt 1: Hostname + SSL flag

STORAGE_ENDPOINT="s3.example.com"
STORAGE_SSL="true"

Result: Get "http://s3.example.com/my-bucket/?location=": dial tcp 10.x.x.x:80

Attempt 2: HTTPS protocol in endpoint

STORAGE_ENDPOINT="https://s3.example.com"
STORAGE_SSL="true"

Result: Endpoint url cannot have fully qualified paths

Attempt 3: Explicit port 443

STORAGE_ENDPOINT="s3.example.com:443"
STORAGE_SSL="true"

Result: Endpoint url cannot have fully qualified paths

Attempt 4: IP address + SSL flag

STORAGE_ENDPOINT="10.x.x.x"
STORAGE_SSL="true"

Result: Get "http://10.x.x.x/my-bucket/?location=": dial tcp 10.x.x.x:80

Attempt 5: Multiple SSL environment variables

STORAGE_ENDPOINT="10.x.x.x"
STORAGE_SSL="true"
STORAGE_SECURE="true"
MINIO_SECURE="true"
MINIO_SSL="true"
S3_USE_SSL="true"
S3_SECURE="true"
SSL_ENABLED="true"
USE_SSL="true"

Result: Still connects to HTTP port 80

Attempt 6: Custom port parameter

STORAGE_ENDPOINT="10.x.x.x"
STORAGE_ENDPOINT_PORT="443"
STORAGE_SSL="true"

Result: Variable ignored, still connects to HTTP port 80

Workaround Attempts (All Failed)

NetworkPolicy Configuration

We configured a Kubernetes NetworkPolicy to explicitly allow egress traffic from Testkube API pods to the S3 endpoint on port 443:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: testkube-api-to-s3
namespace: testkube-namespace
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: testkube-api
policyTypes:

• Egress
  egress:
• to:
  • podSelector: {}
    ports:
  • protocol: TCP
    port: 443
• to:
  • namespaceSelector: {}
    ports:
  • protocol: TCP
    port: 443

Result: NetworkPolicy allows HTTPS traffic, but Testkube still attempts HTTP on port 80

NGINX Proxy Workaround

We attempted to deploy an in-cluster NGINX proxy to convert HTTP requests to HTTPS:

• NGINX listens on port 9000 (HTTP)
• NGINX proxies to S3 endpoint on port 443 (HTTPS)
• Testkube configured with STORAGE_ENDPOINT="nginx-proxy:9000" and STORAGE_SSL="false"

Result: Failed due to additional infrastructure constraints (image pull policies, pod security policies, etc.)

Root Cause Analysis

The MinIO Go SDK appears to:

1. Reject endpoint URLs containing protocol (https://) or explicit ports (:443)
2. Accept plain hostnames/IPs BUT then default to HTTP (port 80) despite secure: true flag
3. Ignore all SSL-related environment variables

This behavior suggests either:

• A bug in how Testkube maps environment variables to MinIO SDK parameters
• A regression in MinIO Go SDK version used by Testkube 2.9.3
• Missing implementation of the useSSL parameter in MinIO client initialization

Impact

• Severity: High — blocks HTTPS-only S3 deployments (common in enterprise environments with strict security policies)
• Workaround: None found (unless storage backend opens HTTP port 80, which is often prohibited by security policies)
• Affected Users: Anyone using S3/Ceph with HTTPS-only configuration in security-restricted environments
• Version Lock-in: Cannot downgrade to older versions due to PostgreSQL support requirement (added in 2.9.0)

Questions

1. Which MinIO Go SDK version is embedded in Testkube 2.9.3?
2. Is the useSSL parameter correctly passed when initializing the MinIO client?
3. Was this working in previous versions (2.8.x with MongoDB)?
4. Is there an undocumented environment variable or configuration option we're missing?
5. Are there plans to support custom S3 port configuration?

Temporary Solution

Currently deploying without S3 storage as a workaround, but this prevents artifact and log persistence.

Additional Context

• NetworkPolicies correctly allow traffic to S3 endpoint on port 443 (verified)
• S3/Ceph endpoint is accessible via HTTPS (verified with curl from within cluster)
• PostgreSQL and NATS connections work correctly
• Only MinIO/S3 connection exhibits this issue
• Enterprise security policies prevent opening HTTP port 80 on S3 storage

Code Reference Request

Could you point us to the code where MinIO client is initialized in cmd/api-server/commons/commons.go? We'd like to verify if the useSSL parameter is correctly set based on the STORAGE_SSL environment variable.

[vsukhin]

thank you @hamzarhaiem let Copilot to check it out

[vsukhin]

@hamzarhaiem does it work for you #7730 ?

[hamzarhaiem]

Hi @vsukhin!

TESTED AND CONFIRMED - THE SSL FIX WORKS PERFECTLY! 🎉

Test environment:

• Testkube API: Built from PR fix: [TKC-5795] ensure MinIO client uses SSL when STORAGE_SSL=true without certificates #7730 (commit 5938545)
• S3 Backend: Ceph S3 (HTTPS-only on port 443, self-signed certificate)
• Config: STORAGE_SSL=true, STORAGE_SKIP_VERIFY=true

Results:

• MinIO client now correctly uses HTTPS
• No more dial tcp :80 errors
• The STORAGE_SSL=true variable is now properly respected
• Certificate verification skip works correctly with STORAGE_SKIP_VERIFY=true
• S3 storage bucket connection established successfully

Evidence from logs:
Before the fix: dial tcp :80 connect connection refused
After the fix: Get https://endpoint/bucket/?location=

The connection now uses HTTPS on port 443 as expected. The storage bucket is accessible. This fix resolves a critical issue for users with HTTPS-only S3 endpoints. Thanks for the incredibly fast turnaround!

[vsukhin]

cool! will release it soon