cdk-custom-resource: Error for Lambda is initializing your function

[fs24-chrter]

Describe the bug

I have a Custom Resource in cdk code that calls a Lambda function.

Works fine most of the time. However, if we don't deploy this stack for a longer time we are seeing the following error message during deployment:
"failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: ERROR: Lambda is initializing your function. It will be ready to invoke shortly."

Seems to me that the lambda is in cold-start mode and a simple internal retry of calling the Lambda would fix the issue. If I rerun the deployment again, it will work.

I also found a Github Action that looks very similar to my error: #20123

But the fix attached to it does not apply to already existing Lambda functions.

I had already a support case open in AWS Support Center and their final advise was to open an issue here.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

The cdk should retry to call the Lambda function until it's ready if this error message comes back.

Current Behavior

Error message:
"failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: ERROR: Lambda is initializing your function. It will be ready to invoke shortly."

Reproduction Steps

const lambdaTrigger = new AwsCustomResource(construct, 'TriggerExistingLambdaCustomResource', {
      timeout: Duration.minutes(5), 
      policy: AwsCustomResourcePolicy.fromStatements([
        new PolicyStatement({
          actions: ['lambda:InvokeFunction'],
          effect: Effect.ALLOW,
          resources: [dbMigrationLambda.functionArn],
        }),
      ]),

      onCreate: {
        service: 'Lambda',
        action: 'invoke',
        parameters: {
          FunctionName: dbMigrationLambda.functionName,
          Payload: JSON.stringify({
            'S3Bucket': bucketName,
            'DatabaseName': databaseName
          }),
        },
        physicalResourceId: PhysicalResourceId.of('invokeLambdaCreate-call-lambda'), 
      },
      onUpdate: {
        service: 'Lambda',
        action: 'invoke',
        parameters: {
          FunctionName: dbMigrationLambda.functionName,
          Payload: JSON.stringify({
            'S3Bucket': bucketName,
            'DatabaseName': databaseName
          }),
        },
        physicalResourceId: PhysicalResourceId.of('invokeLambdaUpdate-call-lambda'),
      },
      installLatestAwsSdk: false,
    });

Possible Solution

Internal retry until the Lambda function is ready if this error message comes back.

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.214.0

AWS CDK CLI version

2.1025.0

Node.js Version

20.19.5

OS

Ubuntu 24.04.3

Language

TypeScript

Language Version

TypeScript 5.8.3

Other information

No response

[pahud]

CloudFormation Stack Deployment
    |
    v
CloudFormation invokes Custom Resource Provider Lambda
    |
    v
AwsCustomResource Handler Lambda (running)
    |
    v
Handler makes SDK call: Lambda.invoke(targetFunction)
    |
    v
Target Lambda is in cold start/initializing
    |
    v
AWS Lambda Service returns error: "Lambda is initializing your function"
    |
    v
Handler Lambda catches this error
    |
    v
Handler Lambda sends FAILED response back to CloudFormation
    |
    v
CloudFormation formats the message you see:
"Received response status [FAILED] from custom resource.
Message returned: ERROR: Lambda is initializing your function"

Hi @fs24-chrter,

Thank you for reporting this issue. Based on the analysis, this appears to be a valid concern about how AwsCustomResource handles transient Lambda initialization errors.

The issue occurs in the custom resource handler which doesn't implement retry logic for Lambda cold start scenarios. When your target Lambda function is in an inactive state and needs initialization, the first invocation fails immediately without retrying.

While this is related to #20123, it's a different issue - that one was about the provider framework Lambda itself going inactive, whereas this is about the target Lambda you're invoking being in cold start.

Potential workarounds until this is fixed:

1. Use ignoreErrorCodesMatching to catch the initialization error (though this isn't ideal as it would suppress the error entirely)
2. Keep your target Lambda warm with scheduled invocations
3. Increase the timeout property (you already have 5 minutes which should be sufficient)

Proposed solution:

The handler should implement retry logic with exponential backoff for transient Lambda errors, specifically:

• Errors containing "Lambda is initializing"
• ResourceNotReadyException
• Other transient Lambda service errors

This would align with AWS SDK best practices for handling transient failures.

Off the top of my head, the change would need to be made in the handler's error handling code to detect and retry on Lambda initialization errors.

[sAchin-680]

The handler should implement retry logic with exponential backoff for transient Lambda errors, specifically:

• Errors containing "Lambda is initializing"
• ResourceNotReadyException
• Other transient Lambda service errors

This would align with AWS SDK best practices for handling transient failures.

Hi @pahud , This makes perfect sense. I’d like to take this up and implement retry logic with exponential backoff in the handler for transient Lambda initialization errors ("Lambda is initializing").

If approved, I’ll start working on a PR for this fix.