etcdserver: request timed out

[tooptoop4]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

anyone see this error once?
Error (exit code 1): etcdserver: request timed out, possibly due to previous leader failure

Version(s)

3.4.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

n/a

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[jomcgi]

We ran into the same issue yesterday, If it's related / you saw similar behaviour I'd love to confirm this!

Hypothesis

EKS triggered etcd defragmentation & argo wf controller crashed because it doesn't tolerate api server outages.

Suspicious API Server storage pattern

Control flow diagram

graph TB
    %% Main Control Structure
    ArgoWF["<b>Argo Workflow Controller</b><br/>────────<br/>Goal: Schedule workflows"]
    API["<b>Kubernetes API Server</b><br/>────────<br/>Goal: Maintain cluster state"]
    Etcd["<b>Etcd</b><br/>────────<br/>Stores: Cluster state data"]
    Cleanup["<b>Resource Controller</b><br/>────────<br/>Goal: Clean up terminated resources"]
    Pods["<b>Workflow Pods & IPs</b><br/>────────<br/>State: Running/Available"]
    
    %% Control Actions
    API -->|"<b>CA1:</b> Write state"| Etcd
    ArgoWF -->|"<b>CA2:</b> Create workflows"| API
    API -->|"<b>CA3:</b> Schedule pods<br/>Allocate IPs"| Pods
    Cleanup -->|"<b>CA4:</b> Delete terminated<br/>Release IPs"| API
    
    %% Feedback
    Etcd -.->|"<b>FB1:</b> State data"| API
    API -.->|"<b>FB2:</b> Health & status"| ArgoWF
    Pods -.->|"<b>FB3:</b> Pod status"| API
    Pods -.->|"<b>FB4:</b> Terminated list"| Cleanup
    
    %% Styling
    classDef controller fill:#bbdefb,stroke:#1976d2,stroke-width:2px,color:#000
    classDef process fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000
    
    class ArgoWF,API,Cleanup controller
    class Etcd,Pods process
    
    %% Line styling  
    linkStyle 0,1,2,3 stroke:#2e7d32,stroke-width:3px
    linkStyle 4,5,6,7 stroke:#1565c0,stroke-width:2px,stroke-dasharray: 5 5
    
    %% Legend
    subgraph Legend ["Suspected Failure Mode"]
        direction TB
        L1["<b>Root Cause:</b> Etcd instability (defragmentation?)"]
        L2["<b>Primary Impact:</b> FB1 fails → API Server unresponsive"]
        L3["<b>Cascading Failures:</b><br/>• FB2 fails → Argo WF crashes (does not tolerate API Server interruption)<br/>• CA4 stops → Cleanup halts → IP exhaustion<br/>• CA2, CA3 blocked → Workflows cannot be scheduled"]
        L4["<b>Detection:</b> Argo WF synthetic test pages on-call"]
        L5["<b>Recovery:</b> Self-healing when Etcd stabilizes"]
        
        L1 --> L2 --> L3 --> L4 --> L5
    end
    
    style Legend fill:#fff9c4,stroke:#f57f17,stroke-width:2px,color:#000
    style L1 fill:#ffcdd2,stroke:#c62828,stroke-width:2px,text-align:left,color:#000
    style L2 fill:#ffccbc,stroke:#d84315,stroke-width:2px,text-align:left,color:#000
    style L3 fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px,text-align:left,color:#000
    style L4 fill:#b3e5fc,stroke:#0277bd,stroke-width:2px,text-align:left,color:#000
    style L5 fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,text-align:left,color:#000

Loading