sam local start-api fails to move /var/rapid/aws-lambda-rie-x86_64 in docker image for serverless function

[llacroix]

Description:

Accessing a lambda using a custom docker image with python awslambdaric fails with:

samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

Steps to reproduce:

Create a serverless function with a custom docker image that uses awslambdaric try to access it through an HttpApi locally.

Observed result:

Invoking Container created from httphandler:latest
Image was not found.
Removing rapid images for repo httphandler
Building image..............
Failed to build Docker Image
NoneType: None
Exception on / [GET]
Traceback (most recent call last):
  File "flask/app.py", line 2447, in wsgi_app
  File "flask/app.py", line 1952, in full_dispatch_request
  File "flask/app.py", line 1821, in handle_user_exception
  File "flask/_compat.py", line 39, in reraise
  File "flask/app.py", line 1950, in full_dispatch_request
  File "flask/app.py", line 1936, in dispatch_request
  File "samcli/local/apigw/local_apigw_service.py", line 357, in _request_handler
  File "samcli/commands/local/lib/local_lambda.py", line 144, in invoke
  File "samcli/lib/telemetry/metric.py", line 230, in wrapped_func
  File "samcli/local/lambdafn/runtime.py", line 177, in invoke
  File "samcli/local/lambdafn/runtime.py", line 88, in create
  File "samcli/local/docker/lambda_container.py", line 94, in __init__
  File "samcli/local/docker/lambda_container.py", line 236, in _get_image
  File "samcli/local/docker/lambda_image.py", line 164, in build
  File "samcli/local/docker/lambda_image.py", line 278, in _build_image
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2022-05-19 01:01:55 127.0.0.1 - - [19/May/2022 01:01:55] "GET / HTTP/1.1" 502 -

Expected result:

Should just work or tell how to fix the issue with moving rapid/aws-lambda-rie

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Fedora
2. sam --version: 1.50.0
3. AWS region: us-east-1

[torresxb1]

Thanks for bringing this to our attention! Could you include a basic sample template, handler, and Dockerfile (and anything else necessary) so that we could reproduce this issue more easily and accurately?

[llacroix]

Yes, I'll give it a look when I'll have time

[mjvirt]

I have just run into this issue and it affects sam local invoke etc.

TL;DR: Lambda functions default to Linux/x86_64 architecture even when run locally with SAM. On Linux for arm/aarch(64) only the Docker Engine/CE is available to install. Unlike for Docker Desktop, CE doesn't include QEMU emulation configuration for other platforms (specifically Linux/x86_64). Third-party binfmt is needed. This on very light testing appears to resolve the issue for SAM.

Longer version:

docker ps -a and info shows the command that fails:

"mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie"

Additional environment details:

OS: Fedora Linux 36 (ARM64) - Parallels VM on Macbook M1
sam --version: SAM CLI, version 1.56.0
AWS region: eu-central-1

It is fairly easy to reproduce. Just create the hello-world sam app via $ sam init in an empty directory:
1 - AWS Quick Start Templates
1 - Hello World Example

$ sam local invoke HelloWorldFunction
Invoking app.lambda_handler (python3.9)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-python3.9
Building image...................
Failed to build Docker Image
NoneType: None
Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

The problem is due to a platform mismatch - attempting to run x86_64 images on arm64 architecture. Ordinarily with Docker Desktop on x86_64 it's possible to emulate other architectures such as arm64.

Per the AWS SAM document here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-linux.html it states:

On Linux, to build and run Lambda functions with a different instruction set architecture than your host machine, you must take additional steps to configure Docker. For example, to run arm64 functions on an x86_64 machine, you can run the following command to configure the Docker daemon: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes.

However, multiarch/qemu-user-static only supports the x86_64 host architecture per https://github.com/multiarch/qemu-user-static/.

I did find this and have managed to get it working for SAM: https://github.com/tonistiigi/binfmt.

[wangzhe3224]

Hi I am running into same issue, any update on the thread please?