Breaking change in how `OPENAI_API_KEY` is injected into workload after PR #304

[mirco-bozzolini]

Hello everyone,

I think that the pull request #304 introduces a breaking change on how the env var OPENAI_API_KEY is added to the workload.

In the default values there's a safe example on how to inject that variable through secrets:

# -- Env vars added to the Open WebUI deployment. Most up-to-date environment variables can be found here: https://docs.openwebui.com/getting-started/env-configuration. Variables can be defined as list or map style.
extraEnvVars: []
  # - name: OPENAI_API_KEY
  #   valueFrom:
  #     secretKeyRef:
  #       name: pipelines-api-key
  #       key: api-key
  # - name: OPENAI_API_KEY
  #   valueFrom:
  #     secretKeyRef:
  #       name: openai-api-key
  #       key: api-key

With tools like External Secret Operator you can safely add that env var to the container without hardcoding it within values.

Also THIS IS the best practice in production environments.

I also aknowledge that if you just want to try-out the service a simple attribute in the values is easier to use.

The problem I have is that with these lines of code in the template workload-manager.yaml OPENAI_API_KEY gets written always in the env field:

        {{- if and .Values.enableOpenaiApi .Values.openaiBaseApiUrl (not .Values.openaiBaseApiUrls) (not .Values.pipelines.enabled) }}
        # If only an OpenAI API value is set, set it to OPENAI_API_BASE_URL
        - name: "OPENAI_API_BASE_URL"
          value: {{ .Values.openaiBaseApiUrl | quote }}
        - name: "OPENAI_API_KEY"
          value: {{ .Values.openaiApiKey | quote }}

This produce the following result if you had previously followed the best practices:

 # If only an OpenAI API value is set, set it to OPENAI_API_BASE_URL
        - name: "OPENAI_API_BASE_URL"
          value: <REDACTED>
        - name: "OPENAI_API_KEY" <--- IT GETS WRITTEN HERE
          value: "0p3n-w3bu!"
          ....
        - name: OPENAI_API_KEY   <--- BUT ALSO HERE
          valueFrom:
            secretKeyRef:
              key: openai_api_key
              name: open-webui-managed-secret

That result in an error since there could not be duplicated env vars.

I tried on using the extraEnvFrom like so:

  extraEnvFrom: 
    # - configMapRef:
    #     name: my-config
    - secretRef:
        name: extra-env-from-secret

But still the OPENAI_API_KEY written by the template in the env field takes precedence.

I think this is the wrong approach.

My proposed solution is to check whether the value of openaiApiKey is empty and set it in the templates only if it not, so this change is not breaking if someone wants to follow security best practices or not.

If you give me a go for this solution I can make the change myself in no time.

[westbrook-ai]

Hey @mirco-bozzolini, thanks for the report. I am starting to lean towards reverting PR #304 since there have been a handful of issues associated with it. I'll do a more detailed dive on this today, but I should have caught that the OpenAI key is explicitly set in env, as the intent is not to force people into bad secret management practices through the patterns in this chart.

[mirco-bozzolini]

Thanks for reaching out to me @0xThresh. An alternative to the revert would be to set the values of the introduced attributes to empty and write them to the workload only if the user has provided values for them.

[westbrook-ai]

@this-is-tobi are you willing to evaluate/ help implement mirco's suggested fix on this one? The issue appears to be valid and we can't easily revert your PR since other PRs have been merged since. Otherwise I'll work on manually reverting it.

[this-is-tobi]

Hey, sorry for the late answer!

I'm not sure I fully understand the issue, @mirco-bozzolini. You said:

That result in an error since there could not be duplicated env vars.

However, Kubernetes can handle multiple definitions of the same env variable, and the last one takes precedence, as mentioned in the Kubernetes documentation:

When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence.

So it should handle variable precedence as follows:

Situation	Effective Value
Same key in envFrom (ConfigMap/Secret) and in env	env value wins
Same key defined multiple times in envFrom (e.g. multiple ConfigMaps/Secrets)	Value from the last envFrom source wins
Same key defined multiple times in env	Value from the last env entry wins

Kubernetes env var precedence tests

The documentation is not very explicit, but we can confirm the precedence behavior by running the following tests:

# Test 1: env beats envFrom
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
stringData:
  MY_VAR: "value-from-secret"
---
apiVersion: v1
kind: Pod
metadata:
  name: test1-env-beats-envfrom
spec:
  containers:
  - name: test
    image: busybox:1.36
    command: ['sleep', '3600']
    envFrom:
    - secretRef:
        name: test-secret
    env:
    - name: MY_VAR
      value: "value-from-env"
EOF

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/test1-env-beats-envfrom --timeout=30s

# Check result (should show "value-from-env")
kubectl exec test1-env-beats-envfrom -- sh -c 'env | grep MY_VAR'

# Cleanup
kubectl delete pod test1-env-beats-envfrom
kubectl delete secret test-secret

# Test 2: Last envFrom wins
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: secret-1
type: Opaque
stringData:
  MY_VAR: "from-secret-1"
---
apiVersion: v1
kind: Secret
metadata:
  name: secret-2
type: Opaque
stringData:
  MY_VAR: "from-secret-2"
---
apiVersion: v1
kind: Pod
metadata:
  name: test2-last-envfrom-wins
spec:
  containers:
  - name: test
    image: busybox:1.36
    command: ['sleep', '3600']
    envFrom:
    - secretRef:
        name: secret-1
    - secretRef:
        name: secret-2
EOF

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/test2-last-envfrom-wins --timeout=30s

# Check result (should show "from-secret-2")
kubectl exec test2-last-envfrom-wins -- sh -c 'env | grep MY_VAR'

# Cleanup
kubectl delete pod test2-last-envfrom-wins
kubectl delete secret secret-1 secret-2

# Test 3: Last env entry wins
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test3-last-env-wins
spec:
  containers:
  - name: test
    image: busybox:1.36
    command: ['sleep', '3600']
    env:
    - name: MY_VAR
      value: "first"
    - name: MY_VAR
      value: "second"
    - name: MY_VAR
      value: "third"
EOF

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/test3-last-env-wins --timeout=30s

# Check result (should show "third")
kubectl exec test3-last-env-wins -- sh -c 'env | grep MY_VAR'

# Cleanup
kubectl delete pod test3-last-env-wins

# Test 4: Complete precedence (env beats envFrom, last wins)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: base-secret
type: Opaque
stringData:
  API_KEY: "from-base-secret"
  DB_URL: "postgres://localhost/db"
---
apiVersion: v1
kind: Secret
metadata:
  name: override-secret
type: Opaque
stringData:
  API_KEY: "from-override-secret"
---
apiVersion: v1
kind: Pod
metadata:
  name: test4-complete
spec:
  containers:
  - name: test
    image: busybox:1.36
    command: ['sleep', '3600']
    envFrom:
    - secretRef:
        name: base-secret
    - secretRef:
        name: override-secret
    env:
    - name: API_KEY
      value: "from-env-final"
    - name: CUSTOM
      value: "first-custom"
    - name: CUSTOM
      value: "second-custom"
EOF

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/test4-complete --timeout=30s

# Check results
echo "=== API_KEY (should be 'from-env-final') ==="
kubectl exec test4-complete -- sh -c 'env | grep API_KEY'
echo ""
echo "=== DB_URL (should be 'postgres://localhost/db') ==="
kubectl exec test4-complete -- sh -c 'env | grep DB_URL'
echo ""
echo "=== CUSTOM (should be 'second-custom') ==="
kubectl exec test4-complete -- sh -c 'env | grep CUSTOM'

# Cleanup
kubectl delete pod test4-complete
kubectl delete secret base-secret override-secret

Expected Results:

Test	Variable	Expected Value	Why
Test 1	MY_VAR	value-from-env	env beats envFrom
Test 2	MY_VAR	from-secret-2	Last envFrom wins
Test 3	MY_VAR	third	Last env entry wins
Test 4	API_KEY	from-env-final	env beats both secrets
Test 4	DB_URL	postgres://localhost/db	Only in base-secret
Test 4	CUSTOM	second-custom	Last env entry wins

Possible Explanation

• If you're having trouble even using extraEnvVars, it might be because of the pipelines.enabled value. When pipelines.enabled is true (which is the default), there are 2 URLs (one for OpenAI and one for the pipeline service), so the variable to use should be OPENAI_API_KEYS (plural) with 2 keys inside (comma-separated) instead of OPENAI_API_KEY (singular).

• You are probably using extraEnvVars as a list which overrode the previous default value with your list, so the final result was only your secret and it worked correctly before. I wanted to keep the default value, so I defined it in openaiApiKey and openaiApiKeys, but now they are not overridden by any extraEnvVars anymore (I think I did a bad choice on this).

• However, I agree that in the current version it is impossible to override the OPENAI_API_KEY and OPENAI_API_KEYS values using the envFrom secret reference because the variables are always defined in the template env section with a default value even with openaiApiKeys: [] or openaiApiKey: "".

Need More Information?

If I've misunderstood your issue, could you please provide a complete example of your setup (including your values.yaml configuration and how you're managing your secrets)? This would help me reproduce the exact problem you're experiencing and ensure I'm addressing the right concern.

Proposed Fix

I just created a PR to fix this behavior and it should now work as before. Just let me know if everything sounds good to you!