Upgrade base of docker-sonic-mgmt to Ubuntu 24.04 (sonic-net#24306)

Why I did it
The base image Ubuntu 20.04 of docker-sonic-mgmt is end of support now. Need to upgrade the base image and the dependent packages to address potential vulnerability issues.

DO NOT MERGE until sonic-net/sonic-mgmt#21045 is merged and cherry-picked to all current active branches.

Work item tracking
Microsoft ADO (number only):
How I did it
Upgraded base image of docker-sonic-mgmt to Ubuntu 24.04.
Upgraded the build environment to bookworm.
Removed the logic of building flag LEGACY_SONIC_MGMT_DOCKER. All the sonic-mgmt branches are using python3 now. It's not necessary to build docker-sonic-mgmt to support python2. And python2 is end of support too.
Version of python packages are not pinned down. This change has pros and cons.
Pros: Keep the docker-sonic-mgmt up to date to avoid future big version leap which could be difficult to handle.
Cons: Community package upgrade could have issue and cause regression.
Version of ansible is pinned to 11.10.0. It's because pytest-ansible does not support the latest ansible due to pytest-ansible's stdout_callback fails on Ansible 2.19 ansible/pytest-ansible#489
Python packages are installed to /opt/venv by following the recommendation of Ubuntu 24.04.
The dash-api package is reverted to install from source code. The dash-api debian package built in bookworm cannot be installed in Ubuntu 24.04 due to dependency issues.
Default user "ubuntu" of Ubuntu 24.04 is removed from the image to avoid potential issues. For details, please refer to Improve setup-container.sh for Ubuntu 24.04 based docker-sonic-mgmt sonic-mgmt#20761. Skipped adding regular user. With this change, the image will only have root user. The setup-container.sh tool in the sonic-mgmt repository can add regular user. Recommend to use the setup-container.sh tool to create sonic-mgmt container.
Improved the code for installing docker and azure-cli.
Remove the build dependency of the ptf debian package. It's because the ptf package is installed by python pip.
How to verify it
After the upgrade, there are some compatibility issues. The compatibility issues are fixed in sonic-net/sonic-mgmt#21045.
I am using that PR to verify that all PR test scripts can pass by using the upgraded docker-sonic-mgmt.

The fix in sonic-net/sonic-mgmt#21045 is backward compatible. Backward compatibility is verified by PR testing of sonic-net/sonic-mgmt#20851

Author: wangxin

.azure-pipelines/docker-sonic-mgmt-mixed.yml

@@ -55,8 +55,8 @@ stages:
         set -xe
         git submodule update --init --recursive -- src/sonic-platform-daemons src/sonic-genl-packet src/sonic-sairedis src/ptf src/sonic-device-data src/sonic-dash-api
 
-        make SONIC_BUILD_JOBS=$(nproc) DEFAULT_CONTAINER_REGISTRY=publicmirror.azurecr.io ENABLE_DOCKER_BASE_PULL=y configure PLATFORM=generic DOCKER_BUILDKIT=0
-        make -f Makefile.work BLDENV=bullseye SONIC_BUILD_JOBS=$(nproc) DEFAULT_CONTAINER_REGISTRY=publicmirror.azurecr.io ENABLE_DOCKER_BASE_PULL=y LEGACY_SONIC_MGMT_DOCKER=n target/docker-sonic-mgmt.gz
+        make NOBUSTER=1 NOBULLSEYE=1 SONIC_BUILD_JOBS=$(nproc) DEFAULT_CONTAINER_REGISTRY=publicmirror.azurecr.io ENABLE_DOCKER_BASE_PULL=y configure PLATFORM=generic DOCKER_BUILDKIT=0
+        make -f Makefile.work BLDENV=bookworm SONIC_BUILD_JOBS=$(nproc) DEFAULT_CONTAINER_REGISTRY=publicmirror.azurecr.io ENABLE_DOCKER_BASE_PULL=y target/docker-sonic-mgmt.gz
         cp target -r $(Build.ArtifactStagingDirectory)/target
         docker load -i target/docker-sonic-mgmt.gz
         docker tag docker-sonic-mgmt $REGISTRY_SERVER/docker-sonic-mgmt:latest

.azure-pipelines/docker-sonic-mgmt.yml

@@ -160,10 +160,6 @@ ifeq ($(ENABLE_DOCKER_BASE_PULL),)
 	override ENABLE_DOCKER_BASE_PULL = n
 endif
 
-ifeq ($(LEGACY_SONIC_MGMT_DOCKER),)
-	override LEGACY_SONIC_MGMT_DOCKER = y
-endif
-
 ifeq ($(CONFIGURED_ARCH),amd64)
 SLAVE_BASE_IMAGE = $(SLAVE_DIR)
 MULTIARCH_QEMU_ENVIRON = n
@@ -584,7 +580,6 @@ SONIC_BUILD_INSTRUCTION :=  $(MAKE) \
                            SONIC_OS_VERSION=$(SONIC_OS_VERSION) \
                            PIP_HTTP_TIMEOUT=$(PIP_HTTP_TIMEOUT) \
                            BUILD_PROCESS_TIMEOUT=$(BUILD_PROCESS_TIMEOUT) \
-                           LEGACY_SONIC_MGMT_DOCKER=$(LEGACY_SONIC_MGMT_DOCKER) \
                            SONIC_PTF_ENV_PY_VER=$(SONIC_PTF_ENV_PY_VER) \
 						   ENABLE_MULTIDB=$(ENABLE_MULTIDB) \
                            $(SONIC_OVERRIDE_BUILD_VARS)