Large amount of time spent in "zipping" when building a pex from a --pex-repository #1675
Description
For the attached lock.txt (superset) and subset.txt requirements, building a PEX using --pex-repository (on an AWS p2x instance inside of Docker) takes ~220s, primarily inside the "zipping" phase:
pex: Zipping PEX file.
pex: Zipping PEX file.: 221244.8ms
The full PEX command to build the subset is:
$HOME/.pyenv/versions/3.10.1/bin/python ./pex \
--tmpdir.tmp \
--jobs 8 \
--python-path .... \
--output-file example.pex \
--no-emit-warnings \
--manylinux manylinux2014 \
--venv prepend \
--requirements-pex local_dists.pex \
--pex-repository default_lockfile.pex \
--interpreter-constraint CPython==3.8.* \
--entry-point ... \
--sources-directory=source_files \
albumentations<2.0.0,>=1.1.0 \
boto3==1.20.24 \
botocore==1.23.24 \
docstring-parser<0.14.0,>=0.13 \
jpeg4py<0.2.0,>=0.1.4 \
jsonargparse[signatures]<5.0.0,>=4.3.1 \
numpy<2.0.0,>=1.22.3 \
nvidia-dali-cuda110>=1.11.0 \
opencv-python<5.0.0,>=4.5.5 \
pytorch-lightning<2.0.0,>=1.5.10 \
requests<3.0.0,>=2.27.1 \
s3fs==2022.2.0 \
setuptools==59.5.0 \
torch<2.0.0,>=1.10.2 \
torchvision<0.12.0,>=0.11.3 \
wandb<0.13.0,>=0.12.11 \
--layout zipapp