Failed to build Release/2.6 (8189a99)

[jeng1220]

问题描述 Issue Description

Build release/2.6 branch but triggers following error if -DWITH_TESTING=ON:

/workspace/paddle/paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/fpA_intB_gemm_template.h:48:10: fatal error: paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/autogen/arch_define.h: No such file or directory
   48 | #include "paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/autogen/arch_define.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

According the error message, it needs files automatically generated by something.
Manually running phi/.../generic_mixed_gemm_kernelLauncher.py triggers another error - FileNotFoundError until I manually created the autogen folder:

$ cd /workspace/paddle/paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm
$ python ./generic_mixed_gemm_kernelLauncher.py

Traceback (most recent call last):
  File "/home/scratch.rjeng_sw/baidu/paddle/paddle/paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/./generic_mixed_gemm_kernelLauncher.py", line 212, in <module>
    with open(header_name, "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'autogen/arch_define.h'

$ mkdir autogen
$ python ./generic_mixed_gemm_kernelLauncher.py # generating needed files

The CMakeLists should cover build process instead of asking users manually creating files and folder.

版本&环境信息 Version & Environment Information

CCACHE_DIR=/workspace/ccache_storage
ARCH_FLAGS="-march=sandybridge -mtune=broadwell"
CXX_FLAGS="$ARCH_FLAGS -Wno-error=range-loop-construct -Wno-error=maybe-uninitialized -Wno-error=switch -faligned-new=32"
CUDA_FLAGS="-t2 --forward-unknown-to-host-compiler -Xfatbin=-compress-all $ARCH_FLAGS -lineinfo"

SKIP_DOWNLOAD_INFERENCE_DATA=ON cmake -Bpaddle/build -Spaddle \
    -GNinja \
    -DINFERENCE_DEMO_INSTALL_DIR=/home/scratch.rjeng_sw/baidu/paddle/test_data \
    -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \
    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_CUDA_FLAGS="$CUDA_FLAGS" \
    -DCUDA_ARCH_NAME=Manual \
    -DCUDA_ARCH_BIN="80" \
    -DWITH_INCREMENTAL_COVERAGE=OFF \
    -DWITH_INFERENCE_API_TEST=OFF \
    -DWITH_DISTRIBUTE=ON \
    -DWITH_COVERAGE=OFF \
    -DWITH_TENSORRT=OFF \
    -DWITH_TESTING=ON \
    -DWITH_CONTRIB=ON \
    -DWITH_ROCM=OFF \
    -DWITH_RCCL=OFF \
    -DWITH_STRIP=ON \
    -DWITH_MKL=OFF \
    -DWITH_AVX=OFF \
    -DWITH_GPU=ON \
    -DWITH_PYTHON=ON \
    -DWITH_CUDNN_FRONTEND=ON \
    -DPY_VERSION=$PYVER \
    -Wno-dev

cmake --build paddle/build -j$((`nproc`))

[jeng1220]

cc @onecatcn for vis

[winter-wang]

You should first remove the two folders "paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/autogen" and "paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/autogen_tmp" , and then re-execute cmake to generate new and correct files。

[jeng1220]

@winter-wang ,
NO, as I mentioned, the folder paddle/phi/kernels/fusion/cutlass/cutlass_kernels/fpA_intB_gemm/autogen doesn't exist before I manaully create it.
The point is that CMake doesn't automatically generate the needed files.
Please clone a new PaddlePaddle, checkpout to 8189a99, and build it with -DWITH_TESTING=ON. You should see the error message.

[jeng1220]

For unknown reason, I cannot reproduce this error now, so close this issue.
@winter-wang ,
This bug only happens when people switches branch.
The step to reproduce:

1. clone the PaddlePaddle
2. checkout to release/2.6
3. run configuration (cmake ..) successfully
4. switch another branch. for example, another private branch based on release/2.6
5. run configuration again. NOW, the cmake will delete the fpA_intB_gemm/autogen because of CMakeLists.txt#L149. However, for unknown reason, it doesn't create the fpA_intB_gemm/autogen later, so the next line, generic_mixed_gemm_kernelLauncher.py will trigger FileNotFoundError

Log:

git clone PaddlePaddle
git checkout release/2.6 # now fpA_intB_gemm/autogen doesn't exist
mkdir build && cd build # create an empty build folder
cmake ..  # run configuration successfully and fpA_intB_gemm/autogen will be generated
cd .. && git branch my_branch && git checkout my_branch # switch another branch
cd build && cmake .. # run configuration again. 
                     # kernels/CMakeLists.txt#L149 deletes fpA_intB_gemm/autogen.
                     # fpA_intB_gemm/autogen doesn't exist
                     # generic_mixed_gemm_kernelLauncher.py triggers FileNotFoundError