[Enhancement][Data Prepper] Dynamic probe configuration

[obouchta]

Is your feature request related to a problem? Please describe.
The pod is being terminated by Kubernetes health probes, particularly during simultaneous startup of multiple pods. This appears to be related to a partition lock issue that delays readiness, causing the probes to fail.

Describe the solution you'd like
Make the probe configuration dynamic to allow adaptation based on runtime conditions or operational needs.

Describe alternatives you've considered
Disable HPA.

Additional context
Liveness probe failed: Get "http://10.105.12.153:4900/list": dial tcp 10.105.12.153:4900: connect: connection refused
Readiness probe failed: Get "http://10.105.12.153:4900/list": dial tcp 10.105.12.153:4900: connect: connection refused

[obouchta]

I attempted to create a Pull Request but encountered an 'Access Denied' error. Could you please apply the changes on my behalf? Thank you.

values.yaml:

# -- Liveness probe configuration
livenessProbe:
  initialDelaySeconds: 2
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 2

# -- Readiness probe configuration
readinessProbe:
  initialDelaySeconds: 0
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3

templates/deployment.yaml:

...
          livenessProbe:
            httpGet:
              path: /list
              port: server
              {{- if not ( empty (.Values.config).ssl ) }}
              scheme: HTTPS
              {{- end }}
            initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
            periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
            timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
            successThreshold: {{ .Values.livenessProbe.successThreshold }}
            failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
          readinessProbe:
            httpGet:
              path: /list
              port: server
              {{- if not ( empty (.Values.config).ssl ) }}
              scheme: HTTPS
              {{- end }}
            initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
            periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
            timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
            successThreshold: {{ .Values.readinessProbe.successThreshold }}
            failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
...

[andrross]

Catch All Triage - 1 2