feat: resolve steps referencing StepActions in concurrently

Avoids unnecessary DeepCopy operations on steps that do not reference a StepAction.

Introduces concurrent resolution of steps that reference StepActions to improve the performance of TaskRun reconciliation, especially when using remote resolvers like git. The key changes include:
- `hasStepRefs` function: A new function that quickly checks if a `TaskSpec` contains any steps referencing `StepActions`. This allows for an early exit if no resolution is needed, avoiding unnecessary work.
- `resolveStepRef` function: This new function encapsulates the logic for resolving a single `StepAction` reference. It handles fetching the remote resource, merging the `StepAction` with the step's specification, and returning the resolved step
- Two-phase resolution: The `GetStepActionsData` function is now split into two distinct phases:
  - Concurrent Resolution: All `StepAction` references are resolved concurrently using an `errgroup`.
  - Sequential Merging: The resolved steps and their provenance are merged into the final step list and the `TaskRun` status sequentially.
- `updateTaskRunProvenance` function: A dedicated function for updating the TaskRun's status with provenance information.

The maximum number of StepActions that can be resolved concurrently is defined by the default config and its `default-step-ref-concurrency-limit` key.

Author: Maximilien-R

config/config-defaults.yaml

@@ -157,3 +157,11 @@ data:
     # This value is used by the sidecar-tekton-log-results container and can be tuned for performance or test scenarios.
     # Example values: "100ms", "500ms", "1s"
     default-sidecar-log-polling-interval: "100ms"
+
+    # default-step-ref-concurrency-limit specifies the concurrency limit for resolving step references.
+    # This setting controls the maximum number of concurrent goroutines used to resolve
+    # step references (`step.ref` fields) simultaneously. This limit acts as a throttle
+    # to prevent overwhelming remote servers (e.g., git providers, OCI registries) or
+    # the Kubernetes API server, especially when a TaskRun contains many steps that
+    # reference StepActions.
+    default-step-ref-concurrency-limit: "5"

docs/additional-configs.md

@@ -34,6 +34,7 @@ installation.
   - [TaskRuns with `imagePullBackOff` Timeout](#taskruns-with-imagepullbackoff-timeout)
   - [Disabling Inline Spec in TaskRun and PipelineRun](#disabling-inline-spec-in-taskrun-and-pipelinerun)
   - [Exponential Backoff for TaskRun and CustomRun Creation](#exponential-backoff-for-taskrun-and-customrun-creation)
+  - [Limiting Step reference concurrency resolution](#limiting-step-reference-concurrency-resolution)
   - [Next steps](#next-steps)
 
 
@@ -782,6 +783,33 @@ If `enable-wait-exponential-backoff` is not set or is set to `"false"`, the cont
 
 **Note:** This feature is especially useful in clusters where webhook services (such as Kyverno, OPA, or custom admission controllers) may be temporarily unavailable or slow to respond.
 
+## Limiting Step reference concurrency resolution
+
+You can control the maximum number of concurrent goroutines that the Tekton controller uses to resolve steps referencing a `StepAction` via the `step.ref` field.
+
+When a `TaskRun` is processed, any step that uses a `ref` to a remote `StepAction` (e.g., one stored in a git repository or an OCI registry) triggers a fetch request. If a `Task` contains many such steps, the controller will attempt to resolve them all in parallel. This can lead to a "thundering herd" problem, potentially overwhelming remote servers, hitting API rate limits, saturating network resources, or placing excessive load on the Kubernetes API server and the Tekton controller itself.
+
+To mitigate this, Tekton Pipelines includes a configurable concurrency limit. By default, a sensible limit is already in place to ensure stability.
+
+#### Default Behavior
+
+If the `default-step-ref-concurrency-limit` key is not set in the `config-defaults` ConfigMap, Tekton Pipelines defaults to a concurrency limit of **5**. This provides a safe, built-in throttle without requiring any initial configuration.
+
+#### Overriding the Default
+
+You can override this default to better suit your environment's capacity (e.g., a high-capacity, self-hosted git server might allow for a higher limit). To change the limit, set the `default-step-ref-concurrency-limit` key in your `config-defaults` `ConfigMap`.
+
+**Example**: To increase the concurrency limit to 20:
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-defaults
+  namespace: tekton-pipelines
+data:
+  default-step-ref-concurrency-limit: "20"
+```
+
 ---
 
 ## Next steps

pkg/apis/config/default.go

@@ -56,6 +56,9 @@ const (
 
 	DefaultSidecarLogPollingInterval = 100 * time.Millisecond
 
+	// DefaultStepRefConcurrencyLimit is the default concurrency limit for resolving step references.
+	DefaultStepRefConcurrencyLimit = 5
+
 	defaultTimeoutMinutesKey                = "default-timeout-minutes"
 	defaultServiceAccountKey                = "default-service-account"
 	defaultManagedByLabelValueKey           = "default-managed-by-label-value"
@@ -70,6 +73,7 @@ const (
 	defaultImagePullBackOffTimeout          = "default-imagepullbackoff-timeout"
 	defaultMaximumResolutionTimeout         = "default-maximum-resolution-timeout"
 	defaultSidecarLogPollingIntervalKey     = "default-sidecar-log-polling-interval"
+	DefaultStepRefConcurrencyLimitKey       = "default-step-ref-concurrency-limit"
 )
 
 // DefaultConfig holds all the default configurations for the config.
@@ -95,6 +99,7 @@ type Defaults struct {
 	// This value is loaded from the 'sidecar-log-polling-interval' key in the config-defaults ConfigMap.
 	// It is used to control the responsiveness and resource usage of the sidecar in both production and test environments.
 	DefaultSidecarLogPollingInterval time.Duration
+	DefaultStepRefConcurrencyLimit   int
 }
 
 // GetDefaultsConfigName returns the name of the configmap containing all
@@ -128,6 +133,7 @@ func (cfg *Defaults) Equals(other *Defaults) bool {
 		other.DefaultImagePullBackOffTimeout == cfg.DefaultImagePullBackOffTimeout &&
 		other.DefaultMaximumResolutionTimeout == cfg.DefaultMaximumResolutionTimeout &&
 		other.DefaultSidecarLogPollingInterval == cfg.DefaultSidecarLogPollingInterval &&
+		other.DefaultStepRefConcurrencyLimit == cfg.DefaultStepRefConcurrencyLimit &&
 		reflect.DeepEqual(other.DefaultForbiddenEnv, cfg.DefaultForbiddenEnv)
 }
 
@@ -143,6 +149,7 @@ func NewDefaultsFromMap(cfgMap map[string]string) (*Defaults, error) {
 		DefaultImagePullBackOffTimeout:    DefaultImagePullBackOffTimeout,
 		DefaultMaximumResolutionTimeout:   DefaultMaximumResolutionTimeout,
 		DefaultSidecarLogPollingInterval:  DefaultSidecarLogPollingInterval,
+		DefaultStepRefConcurrencyLimit:    DefaultStepRefConcurrencyLimit,
 	}
 
 	if defaultTimeoutMin, ok := cfgMap[defaultTimeoutMinutesKey]; ok {
@@ -237,6 +244,14 @@ func NewDefaultsFromMap(cfgMap map[string]string) (*Defaults, error) {
 		tc.DefaultSidecarLogPollingInterval = interval
 	}
 
+	if DefaultStepRefConcurrencyLimit, ok := cfgMap[DefaultStepRefConcurrencyLimitKey]; ok {
+		stepRefConcurrencyLimit, err := strconv.ParseInt(DefaultStepRefConcurrencyLimit, 10, 0)
+		if err != nil {
+			return nil, fmt.Errorf("failed parsing default config %q", DefaultStepRefConcurrencyLimitKey)
+		}
+		tc.DefaultStepRefConcurrencyLimit = int(stepRefConcurrencyLimit)
+	}
+
 	return &tc, nil
 }
 

pkg/apis/config/default_test.go

@@ -47,6 +47,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    time.Duration(5) * time.Second,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 			fileName: config.GetDefaultsConfigName(),
 		},
@@ -69,6 +70,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    0,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 			fileName: "config-defaults-with-pod-template",
 		},
@@ -94,6 +96,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    0,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 		},
 		{
@@ -108,6 +111,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    0,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 		},
 		{
@@ -125,6 +129,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    0,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 		},
 		{
@@ -139,6 +144,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:    time.Duration(15) * time.Second,
 				DefaultMaximumResolutionTimeout:   1 * time.Minute,
 				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:    5,
 			},
 		},
 		{
@@ -153,6 +159,7 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 				DefaultImagePullBackOffTimeout:       0,
 				DefaultMaximumResolutionTimeout:      1 * time.Minute,
 				DefaultSidecarLogPollingInterval:     100 * time.Millisecond,
+				DefaultStepRefConcurrencyLimit:       5,
 			},
 		},
 		{
@@ -203,6 +210,25 @@ func TestNewDefaultsFromConfigMap(t *testing.T) {
 					},
 					"test": {},
 				},
+				DefaultStepRefConcurrencyLimit: 5,
+			},
+		},
+		{
+			expectedError: true,
+			fileName:      "config-defaults-step-ref-concurrency-limit-err",
+		},
+		{
+			expectedError: false,
+			fileName:      "config-defaults-step-ref-concurrency-limit",
+			expectedConfig: &config.Defaults{
+				DefaultStepRefConcurrencyLimit:    10,
+				DefaultTimeoutMinutes:             60,
+				DefaultServiceAccount:             "default",
+				DefaultManagedByLabelValue:        config.DefaultManagedByLabelValue,
+				DefaultMaxMatrixCombinationsCount: 256,
+				DefaultImagePullBackOffTimeout:    0,
+				DefaultMaximumResolutionTimeout:   1 * time.Minute,
+				DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
 			},
 		},
 	}
@@ -228,6 +254,7 @@ func TestNewDefaultsFromEmptyConfigMap(t *testing.T) {
 		DefaultImagePullBackOffTimeout:    0,
 		DefaultMaximumResolutionTimeout:   1 * time.Minute,
 		DefaultSidecarLogPollingInterval:  100 * time.Millisecond,
+		DefaultStepRefConcurrencyLimit:    5,
 	}
 	verifyConfigFileWithExpectedConfig(t, DefaultsConfigEmptyName, expectedConfig)
 }
@@ -414,6 +441,25 @@ func TestEquals(t *testing.T) {
 			},
 			expected: true,
 		},
+		{
+			name: "different default step ref concurrency limit",
+			left: &config.Defaults{
+				DefaultStepRefConcurrencyLimit: 5,
+			},
+			right: &config.Defaults{
+				DefaultStepRefConcurrencyLimit: 10,
+			},
+			expected: false,
+		}, {
+			name: "same default step ref concurrency limit",
+			left: &config.Defaults{
+				DefaultStepRefConcurrencyLimit: 5,
+			},
+			right: &config.Defaults{
+				DefaultStepRefConcurrencyLimit: 5,
+			},
+			expected: true,
+		},
 	}
 
 	for _, tc := range testCases {

pkg/apis/config/testdata/config-defaults-step-ref-concurrency-limit-err.yaml

@@ -0,0 +1,21 @@
+# Copyright 2025 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-defaults
+  namespace: tekton-pipelines
+data:
+  default-step-ref-concurrency-limit: "abc"

pkg/apis/config/testdata/config-defaults-step-ref-concurrency-limit.yaml

@@ -0,0 +1,21 @@
+# Copyright 2025 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-defaults
+  namespace: tekton-pipelines
+data:
+  default-step-ref-concurrency-limit: "10"

pkg/apis/config/testdata/config-defaults.yaml

@@ -22,4 +22,5 @@ data:
   default-service-account: "tekton"
   default-managed-by-label-value: "something-else"
   default-resolver-type: "git"
-  default-imagepullbackoff-timeout: "5s"
+  default-imagepullbackoff-timeout: "5s"
+  default-step-ref-concurrency-limit: "5"