Trilinos: profiler on Frontier

[delcmo]

Hello,

we are trying to use profilers on Frontier (hpctoolkit for instance) and are facing all sort of problems. Has anyone been successful at using profiler with Trilinos-based applications? Any bottleneck we should be aware of.

Thank,

Marco

[cgcgcg]

What kind of issues are you seeing? I used hpctoolkit on another AMD platform.

[delcmo]

The main issue is finding the right modules to compile our Trilinos-based application on Frontier. The original modules we were using are not compatible with the latest version of hpctoolkit. We also run into darshan-runtime not being compatible with profilers in general. Additionally, Triliinos was configured with openmp enabled which seems to be an issue for profilers.

We are looking at the Albany configuration script and environment used on Frontier. It compiles but some of the packages we need are not enabled. We are still trying to compile Trilinos with all packages we need.

I am trying to get ahead of the curve.

Marco

[jhux2]

@delcmo

This is the module environment and configure that we use for Trilinos performance testing on Frontier. I've not tried to build hpctoolkit with this, though.

module reset
module load DefApps/default

module load PrgEnv-amd
module load amd/6.1.3
module load rocm/6.1.3
module swap cray-mpich/8.1.30
module load craype-accel-amd-gfx90a
module load cray-libsci
module load zlib
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load cray-parallel-netcdf

module load ninja

export MPICH_GPU_SUPPORT_ENABLED=1
export TPETRA_ASSUME_GPU_AWARE_MPI=1

export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}

cmake

CMAKE_OPTIONS=(
  #-GNinja
  -D Trilinos_EXTRA_LINK_FLAGS:STRING="${CRAY_XPMEM_POST_LINK_OPTS} -lxpmem ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}"
  -D Trilinos_TEST_CATEGORIES=PERFORMANCE
  -D CMAKE_INSTALL_PREFIX:PATH=${TRILINOS_INSTALL}
  -D CMAKE_BUILD_TYPE:STRING=RELEASE
  -D CMAKE_CXX_STANDARD=17

  -D Trilinos_ENABLE_Fortran:BOOL=OFF
  -D CMAKE_C_COMPILER=mpicc
  -D CMAKE_CXX_COMPILER=mpicxx
  -D CMAKE_C_FLAGS:STRING=""
  -D CMAKE_CXX_FLAGS:STRING="-I${MPICH_DIR}/include"
  -D CMAKE_SKIP_INSTALL_RPATH=TRUE

  -D TPL_ENABLE_MPI:BOOL=ON
  -D MPI_EXEC=srun
  -D MPI_EXEC_NUMPROCS_FLAG:STRING="-c7;--gpus-per-task=1;--gpu-bind=closest;-n"

  -D TPL_ENABLE_SuperLU:BOOL=OFF
  -D ML_ENABLE_SuperLU:BOOL=OFF

  -D TPL_ENABLE_BLAS:BOOL=ON
  -D TPL_BLAS_LIBRARIES:STRING="${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_amd.so"
  -D TPL_ENABLE_LAPACK:BOOL=ON
  -D TPL_LAPACK_LIBRARIES:STRING="${CRAY_LIBSCI_PREFIX_DIR}/lib/libsci_amd.so"

  -D TPL_ENABLE_Boost:BOOL=OFF
  -D TPL_ENABLE_BoostLib:BOOL=OFF

  -D KOKKOSKERNELS_ENABLE_TPL_BLAS:BOOL=ON
  -D KOKKOSKERNELS_ENABLE_TPL_LAPACK:BOOL=ON
  -D KOKKOSKERNELS_ENABLE_TPL_ROCBLAS:BOOL=ON
  -D KOKKOSKERNELS_ENABLE_TPL_ROCSPARSE:BOOL=ON

  -D TPL_ENABLE_Netcdf:BOOL=ON
    -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF_DIR}/include;${PNETCDF_DIR}/include"
    -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF_DIR}/lib;${PNETCDF_DIR}/lib"
  -D TPL_Netcdf_PARALLEL:BOOL=ON

  -D TPL_ENABLE_Pnetcdf:STRING=ON
    -D Pnetcdf_INCLUDE_DIRS:PATH=${PNETCDF_DIR}/include
    -D Pnetcdf_LIBRARY_DIRS:PATH=${PNETCDF_DIR}/lib

  -D TPL_ENABLE_HDF5:STRING=ON
    -D HDF5_INCLUDE_DIRS:PATH=${HDF5_DIR}/include
    -D HDF5_LIBRARY_DIRS:PATH=${HDF5_DIR}/lib\

  -D TPL_ENABLE_ROCBLAS:BOOL=ON
  -D TPL_ENABLE_ROCSOLVER:BOOL=ON
  -D TPL_ENABLE_ROCSPARSE:BOOL=ON

  -D Trilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF
  -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF
  -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF
  -D Trilinos_ENABLE_CXX11:BOOL=ON
  -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF
  -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
  -D Trilinos_ENABLE_SHADOW_WARNINGS=OFF
  -D Trilinos_VERBOSE_CONFIGURE:BOOL=OFF
  -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING=""

  -D Kokkos_ENABLE_HIP:BOOL=ON
  -D Kokkos_ENABLE_EXAMPLES:BOOL=OFF
  -D Trilinos_ENABLE_Epetra:BOOL=OFF
  -D Trilinos_ENABLE_EpetraExt:BOOL=OFF
  -D Kokkos_ENABLE_OPENMP:BOOL=OFF
  -D Kokkos_ENABLE_SERIAL:BOOL=ON
  -D Kokkos_ENABLE_TESTS:BOOL=OFF
  -D Kokkos_ARCH_ZEN3:BOOL=ON

  -D Amesos2_ENABLE_KLU2:BOOL=ON
  -D Intrepid_ENABLE_TESTS:BOOL=OFF
  -D Intrepid2_ENABLE_TESTS:BOOL=OFF
  -D Phalanx_INDEX_SIZE_TYPE:STRING="INT"
  -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="HIP"
  -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF
  -D Tpetra_INST_HIP:BOOL=ON
  -D Tpetra_INST_INT_LONG_LONG:BOOL=ON
  -D Tpetra_INST_INT_INT:BOOL=OFF
  -D Zoltan_ENABLE_ULONG_IDS:BOOL=ON

  -D TPL_ENABLE_Matio:BOOL=OFF
  -D TPL_ENABLE_X11:BOOL=OFF

  -D Trilinos_ENABLE_TESTS:BOOL=OFF

  -D Trilinos_ENABLE_Amesos2:BOOL=ON
  -D Trilinos_ENABLE_Anasazi:BOOL=ON
  -D Trilinos_ENABLE_Belos:BOOL=ON
  -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF
  -D Trilinos_ENABLE_Epetra:BOOL=OFF
  -D Trilinos_ENABLE_EpetraExt:BOOL=OFF
  -D Trilinos_ENABLE_Ifpack2:BOOL=ON
    -D Ifpack2_ENABLE_TESTS:BOOL=ON
    -D Ifpack2_ENABLE_BlockTriDiContainer_Timers:BOOL=ON
  -D Trilinos_ENABLE_Kokkos:BOOL=ON
  -D Trilinos_ENABLE_MueLu:BOOL=ON
    -D MueLu_ENABLE_TESTS:BOOL=ON
  -D Trilinos_ENABLE_OpenMP:BOOL=OFF
  -D Trilinos_ENABLE_Stratimikos:BOOL=ON
  -D Trilinos_ENABLE_Teko:BOOL=ON
  -D Trilinos_ENABLE_Teuchos:BOOL=ON
  -D Trilinos_ENABLE_Thyra:BOOL=ON
  -D Trilinos_ENABLE_ThyraTpetraAdapters:BOOL=ON
  -D Trilinos_ENABLE_Tpetra:BOOL=ON
    -D Tpetra_ENABLE_TESTS:BOOL=ON
  -D Trilinos_ENABLE_Zoltan2:BOOL=ON
  -D Trilinos_ENABLE_Zoltan:BOOL=ON

  -D Kokkos_ENABLE_LIBDL:BOOL=ON
  -D Trilinos_ENABLE_PanzerDofMgr:BOOL=ON
  -D Trilinos_ENABLE_PanzerDiscFE:BOOL=ON
  -D Trilinos_ENABLE_PanzerExprEval:BOOL=ON
  -D Trilinos_ENABLE_PanzerMiniEM:BOOL=ON
    -D PanzerMiniEM_ENABLE_EXAMPLES:BOOL=ON
    -D PanzerMiniEM_ENABLE_TESTS:BOOL=ON
  -D Trilinos_ENABLE_ShyLU_DDFROSch:BOOL=ON
  -D Tpetra_ENABLE_MMM_Timings:BOOL=ON
  -D Tpetra_ASSUME_GPU_AWARE_MPI:BOOL=ON
)

cmake "${CMAKE_OPTIONS[@]}" $TRILINOS_SRC

[delcmo]

@jhux2 is that what is used for Trilinos 17? Are these files available on the GitHub repo?

[jhux2]

is that what is used for Trilinos 17? Are these files available on the GitHub repo?

This is what we use for Trilinos develop, so it should work fine for Trilinos 17. We maintain the performance scripts in an internal repo. The contents don't change too often, however.