fix(custom-resources): add retry logic for Lambda initialization errors in AwsCustomResource

[sai-ray]

Issue # (if applicable)

Closes #35931.

Reason for this change

When AwsCustomResource invokes a target Lambda function that is in a cold start state (inactive), the Lambda service returns an error "Lambda is initializing your function. It will be ready to invoke shortly." The current handler immediately fails without retrying, causing CloudFormation deployments to fail intermittently.

This is particularly problematic for:

• Infrequently deployed stacks
• Lambda functions that go inactive due to low traffic
• Database migration Lambdas triggered during deployments

Description of changes

Added retry logic with exponential backoff to the AwsCustomResource handler to handle Lambda cold start errors. The implementation follows the established pattern from log-retention-handler.

Changes:

• Added retry helper functions to utils.ts:
  • isRetryableError() - Detects Lambda initialization errors
  • calculateDelay() - Exponential backoff with jitter
  • sleep() - Async delay helper
  • makeWithRetry() - Retry wrapper function
• Wrapped apiCall.invoke() in retry logic in index.ts
• Added comprehensive unit tests for retry behavior

Retry Configuration:

• Max retries: 5 attempts
• Initial delay: 1000ms (1 second)
• Max delay: 30000ms (30 seconds)
• Backoff: 2x exponential with jitter

Retryable Errors:

• ResourceNotReadyException error name
• Error message containing "Lambda is initializing" (case-insensitive)

Describe any new or updated permissions being added

N/A - No IAM permission changes. This is a runtime behavior improvement in the Lambda handler.

Description of how you validated changes

• Unit tests: Added 33 new unit tests covering:

  • isRetryableError() - 5 tests for error detection
  • calculateDelay() - 3 tests for backoff calculation
  • makeWithRetry() - 9 tests for retry wrapper behavior
  • Handler retry behavior - 9 tests including:
    • Success on first attempt (no retry needed)
    • Success after retries on Lambda initialization error
    • Success after retries on ResourceNotReadyException
    • Failure after max retries exceeded
    • Non-retryable errors fail immediately
    • ignoreErrorCodesMatching continues to work correctly
    • All request types (Create, Update, Delete) work with retry
    • Case-insensitive error message matching

• Integration tests: Existing integration tests continue to pass unchanged

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

The pull request linter fails with the following errors:

❌ Fixes must contain a change to an integration test file and the resulting snapshot.

If you believe this pull request should receive an exemption, please comment and provide a justification. A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed, add Clarification Request to a comment.

✅ A exemption request has been requested. Please wait for a maintainer's review.

[sai-ray]

Exemption Request

This PR does not include integration test changes because:

1. Runtime-only change: This fix modifies the Lambda handler runtime behavior, not CloudFormation template generation. Integration tests in CDK verify synthesized CloudFormation templates, but this change produces identical templates.

2. No snapshot changes: Since no CloudFormation resources or properties are modified, there are no snapshot changes to capture.

3. Cannot be tested via integration tests: The "Lambda is initializing" error is a transient, timing-dependent condition that occurs when a Lambda function is in cold start state. This cannot be reliably reproduced in integration tests because:

   • It requires the target Lambda to be inactive for an extended period (typically 15+ minutes)
   • The error window is very short (1-10 seconds during initialization)
   • AWS Lambda's cold start behavior is non-deterministic

4. Comprehensive unit test coverage: The PR includes 33 new unit tests that thoroughly cover:

   • Retry logic with mocked Lambda initialization errors
   • Exponential backoff calculation
   • Error detection for retryable vs non-retryable errors
   • All request types (Create, Update, Delete)
   • Edge cases (max retries exceeded, case-insensitive matching)

5. Follows established pattern: This implementation follows the exact same pattern as log-retention-handler which also uses unit tests for retry logic validation rather than integration tests.

The fix is a transparent improvement to the handler's resilience that doesn't affect the CloudFormation contract or require user code changes.

[sai-ray]

This PR was created with Kiro and is here for evaluation purposes. We have not yet decided whether we want to introduce this into the CDK.