Build MOOSE on TACC Vista Cluster - libmesh error

[chunhuizhao478]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

Hi,

I tried to build MOOSE framework using the similar approach I used for previous Lonestar6 installation, but it produces errors when install libmesh:

/opt/apps/gcc/13.2.0/lib/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld: /opt/apps/ucx/1.17.0/lib/libucp.so: undefined reference to `ucs_cpu_get_memcpy_bw'
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:551: ncgen3] Error 1
make[4]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2/ncgen3'
make[3]: *** [Makefile:695: all-recursive] Error 1
make[3]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[2]: *** [Makefile:540: all] Error 2
make[2]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[1]: *** [Makefile:1058: all-recursive] Error 1
make[1]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib'
make: *** [Makefile:35270: all-recursive] Error 1
-bash: return: can only `return' from a function or sourced script

The build script I tried to follow is:

module purge
module load gcc/13.2.0
module load cuda/13.0
module load eigen/3.4.0
module load hdf5/1.14.6
module load netcdf/4.9.2
module load cmake/3.29.5
module load openmpi/5.0.5
echo $CC $CXX $FC $F90 $F77
export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77
build petsc, libmesh follows website

I asked TACC consultant, below is the response:

Can you confirm that moose and libmesh are compatible with aarch64? The
documentation I'm seeing for moose only mentions Apple silicon ARM, but that's
not quite the same as the architecture on Vista. Also, is the petsc you
installed something specific that wasn't covered with the petsc module?

Any suggestions are welcome, thanks!

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[GiudGiud]

Hello

Yes MOOSE builds fine or aarch64.
Can you please run the diagnostics script in moose/scripts ?
And can you please locate libucs.so and libucp.so on your system? This is pointing to an issue with MPI / these communication protocols that this MPI points to

Note: if you also end up trying another MPI compiler, you do need to start from scratch again

[GiudGiud]

you have netcdf in your path and LD_LIBRARY_PATH, and it's erroring while installing it. Cant be a coincidence. Try deleting that from the paths

actually just remove

module load netcdf/4.9.2

[chunhuizhao478]

Hi @GiudGiud, I have tried your suggestions and rebuild petsc and libmesh, however the problem persists, I also show the module list below

make[4]: Entering directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2/ncgen3'
  CC       main.o
  CC       load.o
  CC       escapes.o
  CC       getfill.o
  CC       init.o
  CC       genlib.o
  CC       ncgentab.o
  CCLD     ncgen3
/opt/apps/gcc/13.2.0/lib/gcc/aarch64-unknown-linux-gnu/13.2.0/../../../../aarch64-unknown-linux-gnu/bin/ld: /opt/apps/ucx/1.17.0/lib/libucp.so: undefined reference to `ucs_cpu_get_memcpy_bw'
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:551: ncgen3] Error 1
make[4]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2/ncgen3'
make[3]: *** [Makefile:695: all-recursive] Error 1
make[3]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[2]: *** [Makefile:540: all] Error 2
make[2]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib/netcdf/netcdf-c-4.6.2'
make[1]: *** [Makefile:1058: all-recursive] Error 1
make[1]: Leaving directory '/home1/10024/zhaochun/projects/moose/libmesh/build/contrib'
make: *** [Makefile:35270: all-recursive] Error 1
login1.vista(331)$ module list

Currently Loaded Modules:
  1) gcc/13.2.0   2) cuda/13.0 (g)   3) eigen/3.4.0   4) hdf5/1.14.6   5) cmake/3.29.5   6) openmpi/5.0.5   7) ucx/1.17.0

  Where:
   g:  built for GPU



login1.vista(332)$

[GiudGiud]

is the ucx module loaded when you load module load openmpi/5.0.5

do you have other options for openmpi?

[chunhuizhao478]

Hi @GiudGiud, yes when I load openmpi/5.0.5, the ucx/1.9.0 is automatically loaded, I have tried also swap to ucx/1.7.0, but no luck.

Other openmpi options:

(moose-dev) login1.vista(398)$ module spider openmpi

----------------------------------------------------------------------------------------------------------------------------------------------------------
  openmpi:
----------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      MPI implementation

     Versions:
        openmpi/5.0.3
        openmpi/5.0.4
        openmpi/5.0.5_nvc249
        openmpi/5.0.5
        openmpi/5.0.7
        openmpi/5.0.8

----------------------------------------------------------------------------------------------------------------------------------------------------------

[GiudGiud]

I don't know of a better one. But loading others could be a way to find a compatible ucx

[chunhuizhao478]

Also I think miniforge build is not successful on this ARM cluster (this ARM for Mac is not compatible), I'm working on the suggestions by @milljm in #30775, it fails at petsc build:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
bash Miniforge3-Linux-aarch64.sh -b -p ~/miniforge
export PATH=$HOME/miniforge/bin:$PATH
conda create -n moose-dev gcc=12 gxx=12 gfortran=12 cmake=3 mpi mpich hdf5="*=mpi_mpich_*" bison flex automake libtirpc zfp autoconf libtool make zlib m4 python=3.13 mpich-mpicc mpich-mpicxx mpich-mpifort
conda activate moose-dev
cd projects/
git clone https://github.com/idaholab/moose.git
cd moose/
git checkout master
cd scripts/
./update_and_rebuild_petsc.sh <- error
./update_and_rebuild_libmesh.sh
./update_and_rebuild_wasp.sh
cd ../test
make
./run_tests

*********************************************************************************************
           UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
---------------------------------------------------------------------------------------------
                                 Error running make on  HDF5
*********************************************************************************************
  File "/home1/10024/zhaochun/projects/moose/petsc/config/configure.py", line 461, in petsc_configure
    framework.configure(out = sys.stdout)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1468, in configure
    self.processChildren()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1456, in processChildren
    self.serialEvaluation(self.childGraph)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/framework.py", line 1431, in serialEvaluation
    child.configure()
    ~~~~~~~~~~~~~~~^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 1402, in configure
    self.executeTest(self.configureLibrary)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/base.py", line 138, in executeTest
    ret = test(*args,**kargs)
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/packages/HDF5.py", line 92, in configureLibrary
    config.package.GNUPackage.configureLibrary(self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 1082, in configureLibrary
    for location, directory, lib, incl in self.generateGuesses():
                                          ~~~~~~~~~~~~~~~~~~~~^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 615, in generateGuesses
    d = self.checkDownload()
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 749, in checkDownload
    return self.getInstallDir()
           ~~~~~~~~~~~~~~~~~~^^
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 551, in getInstallDir
    installDir = self.Install()
  File "/home1/10024/zhaochun/projects/moose/petsc/config/BuildSystem/config/package.py", line 2132, in Install
    raise RuntimeError('Error running make on  '+self.PACKAGE)
================================================================================
Finishing configure run at Tue, 02 Dec 2025 11:04:52 -0600
================================================================================
There was an error. Exiting...

[GiudGiud]

You're downloading hdf5 with conda, not sure why it should PETSc is installing HDF5. You could specify the various HDF5 environment variables (HDF5 ROOT, DIR, ?) to avoid that

[milljm]

There should be more information in that configure.log file as well.

[chunhuizhao478]

This is a memory issue, I set export MOOSE_JOBS=6

[chunhuizhao478]

Hi @GiudGiud, @milljm, I have successfully build moose on Vista (ARM Linux) using conda (and Claude ai), below is the script. There are some steps (like step 4) not straightforward. Maybe you will have some ideas to improve it.

For the initial attempt using system loaded modules, I haven't made it work.

#!/bin/bash
#===============================================================================
# MOOSE Build Script for TACC Vista (ARM64/Grace Hopper)
# 
# This script builds MOOSE framework from scratch using conda environment
# with MPICH on TACC Vista cluster.
#
# Prerequisites:
#   - Miniforge/Miniconda installed
#   - Internet access for conda packages and git submodules
#
# Usage:
#   chmod +x build_moose_vista.sh
#   ./build_moose_vista.sh
#
# Author: Generated from debugging session
# Date: December 2024
#===============================================================================

set -e  # Exit on error

#-------------------------------------------------------------------------------
# Configuration
#-------------------------------------------------------------------------------
MOOSE_DIR=${MOOSE_DIR:-/home1/10024/zhaochun/projects/moose}
CONDA_ENV_NAME=${CONDA_ENV_NAME:-moose-dev}
MOOSE_JOBS=${MOOSE_JOBS:-4}

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

log_info() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

log_warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

log_error() {
    echo -e "${RED}[ERROR]${NC} $1"
}

#-------------------------------------------------------------------------------
# Step 1: Purge modules to avoid conflicts
#-------------------------------------------------------------------------------
log_info "Step 1: Purging system modules..."
module purge 2>/dev/null || true

#-------------------------------------------------------------------------------
# Step 2: Create/Update conda environment
#-------------------------------------------------------------------------------
log_info "Step 2: Setting up conda environment..."

# Initialize conda
source /home1/10024/zhaochun/miniforge/etc/profile.d/conda.sh

# Create environment if it doesn't exist
if ! conda env list | grep -q "^${CONDA_ENV_NAME} "; then
    log_info "Creating conda environment: ${CONDA_ENV_NAME}"
    conda create -n ${CONDA_ENV_NAME} -y python=3.11
fi

# Activate environment
conda activate ${CONDA_ENV_NAME}
log_info "Activated conda environment: ${CONDA_ENV_NAME}"

#-------------------------------------------------------------------------------
# Step 3: Install required conda packages
#-------------------------------------------------------------------------------
log_info "Step 3: Installing conda packages..."

conda install -c conda-forge -y \
    mpich \
    mpich-mpicc \
    mpich-mpicxx \
    mpich-mpifort \
    cmake \
    make \
    git \
    pyyaml \
    xz \
    libtirpc \
    "hdf5=*=mpi_mpich_*"

log_info "Conda packages installed successfully"

#-------------------------------------------------------------------------------
# Step 4: Patch tirpc header (fix bindresvport conflict)
#-------------------------------------------------------------------------------
log_info "Step 4: Patching tirpc header..."

TIRPC_HEADER="$CONDA_PREFIX/include/tirpc/rpc/rpc.h"
if [ -f "$TIRPC_HEADER" ]; then
    # Backup original
    if [ ! -f "${TIRPC_HEADER}.bak" ]; then
        cp "$TIRPC_HEADER" "${TIRPC_HEADER}.bak"
    fi
    
    # Check if already patched
    if grep -q "bindresvport.*__THROW" "$TIRPC_HEADER"; then
        log_info "tirpc header already patched"
    else
        # Patch: add __THROW to match system declaration
        sed -i 's/extern int bindresvport(int, struct sockaddr_in \*);/extern int bindresvport(int, struct sockaddr_in *) __THROW;/' "$TIRPC_HEADER"
        log_info "tirpc header patched successfully"
    fi
else
    log_error "tirpc header not found at $TIRPC_HEADER"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 5: Set environment variables
#-------------------------------------------------------------------------------
log_info "Step 5: Setting environment variables..."

export CC=mpicc
export CXX=mpicxx
export FC=mpifort
export F77=mpifort
export F90=mpifort
export CPPFLAGS="-I$CONDA_PREFIX/include/tirpc"
export LDFLAGS="-L$CONDA_PREFIX/lib -ltirpc"
export MOOSE_JOBS=${MOOSE_JOBS}
export MOOSE_DIR=${MOOSE_DIR}

log_info "Environment variables set:"
log_info "  CC=$CC"
log_info "  CXX=$CXX"
log_info "  MOOSE_DIR=$MOOSE_DIR"
log_info "  MOOSE_JOBS=$MOOSE_JOBS"

#-------------------------------------------------------------------------------
# Step 6: Clone MOOSE repository
#-------------------------------------------------------------------------------
log_info "Step 6: Setting up MOOSE repository..."

if [ ! -d "$MOOSE_DIR" ]; then
    mkdir -p "$(dirname $MOOSE_DIR)"
    cd "$(dirname $MOOSE_DIR)"
    git clone https://github.com/idaholab/moose.git
    cd moose
    git checkout master
else
    log_info "MOOSE directory exists, updating..."
    cd "$MOOSE_DIR"
    git fetch origin
fi

cd "$MOOSE_DIR"
git submodule update --init --recursive

#-------------------------------------------------------------------------------
# Step 7: Build PETSc
#-------------------------------------------------------------------------------
log_info "Step 7: Building PETSc (this may take 30-60 minutes)..."

cd "$MOOSE_DIR"

# Clean previous build
if [ -d "petsc/arch-moose" ]; then
    log_info "Cleaning previous PETSc build..."
    git clean -xfd petsc
fi

# Build PETSc
./scripts/update_and_rebuild_petsc.sh

# Verify PETSc build
if [ -f "$MOOSE_DIR/petsc/arch-moose/lib/libpetsc.so" ]; then
    log_info "PETSc built successfully"
else
    log_error "PETSc build failed"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 8: Build libMesh
#-------------------------------------------------------------------------------
log_info "Step 8: Building libMesh (this may take 20-40 minutes)..."

cd "$MOOSE_DIR"

# Clean previous build
if [ -d "libmesh/build" ]; then
    log_info "Cleaning previous libMesh build..."
    git clean -xfd libmesh
fi

# Build libMesh
./scripts/update_and_rebuild_libmesh.sh 

# Verify libMesh build with MPI
if grep -q "#define LIBMESH_HAVE_MPI 1" "$MOOSE_DIR/libmesh/installed/include/libmesh/libmesh_config.h"; then
    log_info "libMesh built successfully with MPI support"
else
    log_error "libMesh build failed or MPI not enabled"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 9: Build MOOSE framework and test
#-------------------------------------------------------------------------------
log_info "Step 9: Building MOOSE framework..."

cd "$MOOSE_DIR"

# Build framework
make -j${MOOSE_JOBS}

# Build test application
cd test
make -j${MOOSE_JOBS}

# Verify build
if [ -x "$MOOSE_DIR/test/moose_test-opt" ]; then
    log_info "MOOSE test application built successfully"
else
    log_error "MOOSE test build failed"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 10: Create environment setup script
#-------------------------------------------------------------------------------
log_info "Step 10: Creating environment setup script..."

cat > "$HOME/setup_moose_vista.sh" << 'SETUP_EOF'
#!/bin/bash
#===============================================================================
# MOOSE Environment Setup Script for TACC Vista
# Source this script before running MOOSE applications
# Usage: source ~/setup_moose_vista.sh
#===============================================================================

# Purge system modules to avoid conflicts
module purge 2>/dev/null || true

# Initialize and activate conda
source /home1/10024/zhaochun/miniforge/etc/profile.d/conda.sh

conda activate moose-dev

# Set compiler environment
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
export F77=mpifort
export F90=mpifort

# Set tirpc paths (required for XDR support)
export CPPFLAGS="-I$CONDA_PREFIX/include/tirpc"
export LDFLAGS="-L$CONDA_PREFIX/lib -ltirpc"

# MOOSE environment
export MOOSE_DIR=/home1/10024/zhaochun/projects/moose
export MOOSE_JOBS=4

# PETSc environment
export PETSC_DIR=$MOOSE_DIR/petsc
export PETSC_ARCH=arch-moose

# Add MOOSE to PATH (optional)
export PATH=$MOOSE_DIR/test:$PATH

echo "MOOSE environment loaded for Vista"
echo "  MOOSE_DIR: $MOOSE_DIR"
echo "  CONDA_PREFIX: $CONDA_PREFIX"
echo "  MPI: $(which mpicc)"
SETUP_EOF

chmod +x "$HOME/setup_moose_vista.sh"

#-------------------------------------------------------------------------------
# Done
#-------------------------------------------------------------------------------
log_info "==============================================="
log_info "MOOSE build completed successfully!"
log_info "==============================================="
log_info ""
log_info "To use MOOSE in future sessions, run:"
log_info "  source ~/setup_moose_vista.sh"
log_info ""
log_info "To test your installation:"
log_info "  cd $MOOSE_DIR/test"
log_info "  ./run_tests -j4"
log_info ""
log_info "To build your own application:"
log_info "  source ~/setup_moose_vista.sh"
log_info "  cd /path/to/your/app"
log_info "  make -j4"

[chunhuizhao478]

Hi @GiudGiud, I do encounter the issue when I use more nodes, the initialization stage for the code halt forever, that might be the reason, I will try to use system MPI and rebuild the conda environment

[chunhuizhao478]

Initializing
   Updating Because Mesh Changed
     Updating Mesh....                                                                  [^[[33m 29.38 s^[[39m] [^[[33m 3835 MB^[[39m]
   Finished Updating Because Mesh Changed                                               [^[[33m 29.53 s^[[39m] [^[[33m 3842 MB^[[39m]
   Updating Because Mesh Changed
     Updating Mesh......                                                                [^[[33m 41.08 s^[[39m] [^[[33m 3861 MB^[[39m]
   Finished Updating Because Mesh Changed                                               [^[[33m 41.19 s^[[39m] [^[[33m 3861 MB^[[39m]
   Initializing Equation Systems.....................................................................................................................................................................................................................................................................................................

The initializing equation is halt forever, I wonder what does this step actually doing?

[GiudGiud]

probably stuck in parallel communication. This is with the conda mpi used for installation right?

[chunhuizhao478]

Yes, that's correct! I have rebuild using system MPI and GCC, the problem turns out to be gcc/13.2.0 is wrongly located in gcc14 folder on Vista (not sure the reason), but after using gcc/14.2.0, the problem of libmesh is solved, and the previous parallel communication error also resolved, below is the full script for others reference (ARM Linux Cluster):

#!/bin/bash
#===============================================================================
# MOOSE Build Script for TACC Vista (ARM64) with System MPI
# 
# This script builds MOOSE using system GCC and OpenMPI instead of conda MPI
# to avoid launcher compatibility issues on Vista's network fabric.
#
# Usage:
#   1. Make executable: chmod +x build_moose_vista_sysmpi.sh
#   2. Run: ./build_moose_vista_sysmpi.sh
#
# Prerequisites:
#   - Miniforge/conda installed at /home1/10024/zhaochun/miniforge
#   - MOOSE cloned at /home1/10024/zhaochun/projects/moose
#
# Author: Generated for Vista build
# Date: December 2025
#===============================================================================

set -e  # Exit on error

#-------------------------------------------------------------------------------
# Configuration - Modify these paths as needed
#-------------------------------------------------------------------------------
MINIFORGE_PATH="/home1/10024/zhaochun/miniforge"
MOOSE_DIR="/home1/10024/zhaochun/projects/moose"
CONDA_ENV_NAME="moose-sys-mpi"
MOOSE_JOBS=4  # Keep low for login node; increase on compute node

#-------------------------------------------------------------------------------
# Color output
#-------------------------------------------------------------------------------
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }

#-------------------------------------------------------------------------------
# Step 1: Purge modules and load system MPI stack
#-------------------------------------------------------------------------------
log_info "Step 1: Loading system modules..."

module purge
module load gcc/14.2.0
module load openmpi/5.0.5

log_info "Loaded modules:"
module list

#-------------------------------------------------------------------------------
# Step 2: Install ARM-Structure Miniforge
#-------------------------------------------------------------------------------
log_info "Step 2: Install ARM Miniforge..."

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh

export PATH=$HOME/miniforge/bin:$PATH

conda init --all
exit

#-------------------------------------------------------------------------------
# Step 3: Create conda environment (if not exists)
#-------------------------------------------------------------------------------
log_info "Step 3: Setting up conda environment '${CONDA_ENV_NAME}'..."

if conda env list | grep -q "^${CONDA_ENV_NAME} "; then
    log_warn "Conda environment '${CONDA_ENV_NAME}' already exists."
    read -p "Recreate it? (y/n) " -n 1 -r
    echo
    if [[ $REPLY =~ ^[Yy]$ ]]; then
        conda env remove -n ${CONDA_ENV_NAME} -y
        conda create -n ${CONDA_ENV_NAME} python=3.11 -y
    fi
else
    log_info "Creating new conda environment..."
    conda create -n ${CONDA_ENV_NAME} python=3.11 -y
fi

conda activate ${CONDA_ENV_NAME}

#-------------------------------------------------------------------------------
# Step 4: Install conda dependencies (NO MPI packages!)
#-------------------------------------------------------------------------------
log_info "Step 4: Installing conda dependencies (without MPI)..."

conda install -c conda-forge \
    cmake \
    make \
    git \
    pyyaml \
    xz \
    libtirpc \
    packaging \
    -y

# Verify no MPI packages installed
if conda list | grep -qE "^(mpich|openmpi|mpi) "; then
    log_error "MPI packages found in conda! Remove them:"
    log_error "conda remove mpich openmpi mpi --force"
    exit 1
fi

log_info "Conda dependencies installed successfully."

#-------------------------------------------------------------------------------
# Step 5: Patch tirpc header (bindresvport exception specifier)
#-------------------------------------------------------------------------------
log_info "Step 5: Patching tirpc header..."

TIRPC_HEADER="${CONDA_PREFIX}/include/tirpc/rpc/rpc.h"

if [ -f "$TIRPC_HEADER" ]; then
    if grep -q "bindresvport.*__THROW" "$TIRPC_HEADER"; then
        log_info "tirpc header already patched."
    else
        sed -i 's/extern int bindresvport(int, struct sockaddr_in \*);/extern int bindresvport(int, struct sockaddr_in *) __THROW;/' \
            "$TIRPC_HEADER"
        log_info "tirpc header patched successfully."
    fi
else
    log_error "tirpc header not found at $TIRPC_HEADER"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 6: Set environment variables
#-------------------------------------------------------------------------------
log_info "Step 6: Setting environment variables..."

# Compilers (use system MPI wrappers)
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
export F77=mpifort
export F90=mpifort

# MOOSE paths
export MOOSE_DIR=${MOOSE_DIR}
export MOOSE_JOBS=${MOOSE_JOBS}

# tirpc and UCX paths
export CPPFLAGS="-I${CONDA_PREFIX}/include/tirpc -I/opt/apps/ucx/1.19.1/include"
export LDFLAGS="-L/opt/apps/ucx/1.19.1/lib -L${CONDA_PREFIX}/lib -ltirpc -Wl,-rpath,/opt/apps/ucx/1.19.1/lib"
export LD_LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:${LD_LIBRARY_PATH}
export LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:${LIBRARY_PATH}

# Verify system MPI is being used
log_info "Checking MPI configuration..."
which mpicc
mpicc --version | head -1

MPI_PATH=$(which mpicc)
if [[ "$MPI_PATH" == *"conda"* ]]; then
    log_error "mpicc is from conda! Should be system MPI."
    exit 1
fi
log_info "Using system MPI: $MPI_PATH"

#-------------------------------------------------------------------------------
# Step 7: Clean previous builds
#-------------------------------------------------------------------------------
log_info "Step 7: Cleaning previous builds..."

cd ${MOOSE_DIR}

if [ -d "petsc" ]; then
    log_info "Cleaning PETSc..."
    cd petsc && git clean -xfd && cd ..
fi

if [ -d "libmesh" ]; then
    log_info "Cleaning libMesh..."
    cd libmesh && git clean -xfd && cd ..
fi

#-------------------------------------------------------------------------------
# Step 8: Build PETSc
#-------------------------------------------------------------------------------
log_info "Step 8: Building PETSc (this takes 30-60 minutes)..."

cd ${MOOSE_DIR}
./scripts/update_and_rebuild_petsc.sh

if [ $? -eq 0 ]; then
    log_info "PETSc built successfully!"
else
    log_error "PETSc build failed!"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 9: Build libMesh
#-------------------------------------------------------------------------------
log_info "Step 9: Building libMesh (this takes 20-40 minutes)..."

cd ${MOOSE_DIR}
./scripts/update_and_rebuild_libmesh.sh

if [ $? -eq 0 ]; then
    log_info "libMesh built successfully!"
else
    log_error "libMesh build failed!"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 10: Build WASP (if exists)
#-------------------------------------------------------------------------------
if [ -d "${MOOSE_DIR}/scripts" ] && grep -q "wasp" ${MOOSE_DIR}/scripts/update_and_rebuild_wasp.sh 2>/dev/null; then
    log_info "Step 10: Building WASP..."
    cd ${MOOSE_DIR}
    ./scripts/update_and_rebuild_wasp.sh
    
    if [ $? -eq 0 ]; then
        log_info "WASP built successfully!"
    else
        log_warn "WASP build failed (may not be required)."
    fi
else
    log_info "Step 10: WASP build script not found, skipping..."
fi

#-------------------------------------------------------------------------------
# Step 11: Build MOOSE Framework
#-------------------------------------------------------------------------------
log_info "Step 11: Building MOOSE Framework..."

cd ${MOOSE_DIR}/framework
make clobber 2>/dev/null || true
make -j${MOOSE_JOBS}

if [ $? -eq 0 ]; then
    log_info "MOOSE Framework built successfully!"
else
    log_error "MOOSE Framework build failed!"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 12: Build MOOSE Test
#-------------------------------------------------------------------------------
log_info "Step 12: Building MOOSE Test..."

cd ${MOOSE_DIR}/test
make clobber 2>/dev/null || true
make -j${MOOSE_JOBS}

if [ $? -eq 0 ]; then
    log_info "MOOSE Test built successfully!"
else
    log_error "MOOSE Test build failed!"
    exit 1
fi

#-------------------------------------------------------------------------------
# Step 13: Verify build
#-------------------------------------------------------------------------------
log_info "Step 13: Verifying build..."

# Check for MPI support in libmesh
if grep -q "LIBMESH_HAVE_MPI 1" ${MOOSE_DIR}/libmesh/installed/include/libmesh/libmesh_config.h; then
    log_info "✓ libMesh has MPI support"
else
    log_error "✗ libMesh missing MPI support!"
fi

# Check executable exists
if [ -f "${MOOSE_DIR}/test/moose_test-opt" ]; then
    log_info "✓ moose_test-opt executable created"
else
    log_error "✗ moose_test-opt not found!"
fi

# Check PETSc UCX linking
log_info "Checking PETSc UCX linking..."
ldd ${MOOSE_DIR}/petsc/arch-moose/lib/libpetsc.so | grep ucx

#-------------------------------------------------------------------------------
# Step 14: Create setup script for future sessions
#-------------------------------------------------------------------------------
log_info "Step 14: Creating setup script..."

cat > ~/setup_moose_vista_sysmpi.sh << 'EOF'
#!/bin/bash
# MOOSE Vista Setup Script (System MPI)
# Source this script to set up MOOSE environment:
#   source ~/setup_moose_vista_sysmpi.sh

module purge
module load gcc/14.2.0
module load openmpi/5.0.7

source /home1/10024/zhaochun/miniforge/etc/profile.d/conda.sh
conda activate moose-sys-mpi

export MOOSE_DIR=/home1/10024/zhaochun/projects/moose
export PETSC_DIR=$MOOSE_DIR/petsc
export PETSC_ARCH=arch-moose

export CPPFLAGS="-I$CONDA_PREFIX/include/tirpc -I/opt/apps/ucx/1.19.1/include"
export LDFLAGS="-L/opt/apps/ucx/1.19.1/lib -L$CONDA_PREFIX/lib -ltirpc -Wl,-rpath,/opt/apps/ucx/1.19.1/lib"
export LD_LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:$LIBRARY_PATH

export CC=mpicc
export CXX=mpicxx
export FC=mpifort

echo "MOOSE environment loaded (System MPI)"
echo "MOOSE_DIR: $MOOSE_DIR"
EOF

chmod +x ~/setup_moose_vista_sysmpi.sh
log_info "Setup script created: ~/setup_moose_vista_sysmpi.sh"

#-------------------------------------------------------------------------------
# Done!
#-------------------------------------------------------------------------------
echo ""
echo "==============================================================================="
log_info "MOOSE build completed successfully!"
echo "==============================================================================="
echo ""
echo "To use MOOSE in future sessions:"
echo "  source ~/setup_moose_vista_sysmpi.sh"
echo ""
echo "To build your application (e.g., FARMS):"
echo "  cd /path/to/farms"
echo "  make clobber"
echo "  make -j4"
echo ""
echo "Job script template:"
echo "  #!/bin/bash"
echo "  #SBATCH -p gg"
echo "  #SBATCH -N 40"
echo "  #SBATCH -n 200"
echo "  #SBATCH -t 48:00:00"
echo "  #SBATCH -A EAR20006"
echo "  "
echo "  module purge"
echo "  module load gcc/14.2.0"
echo "  module load openmpi/5.0.7"
echo "  "
echo "  source /home1/10024/zhaochun/miniforge/etc/profile.d/conda.sh"
echo "  conda activate moose-sys-mpi"
echo "  "
echo "  ibrun ./your_app-opt -i input.i"
echo ""

export CPPFLAGS="-I$CONDA_PREFIX/include/tirpc -I/opt/apps/ucx/1.19.1/include"
export LDFLAGS="-L/opt/apps/ucx/1.19.1/lib -L$CONDA_PREFIX/lib -ltirpc -Wl,-rpath,/opt/apps/ucx/1.19.1/lib"
export LD_LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/opt/apps/ucx/1.19.1/lib:$LIBRARY_PATH
export LD_PRELOAD=/opt/apps/ucx/1.19.1/lib/libucs.so

[GiudGiud]

the problem turns out to be gcc/13.2.0 is wrongly located in gcc14 folder on Vista (not sure the reason)

The folder nesting is often used by HPC admins to indicate which compiler they used to install it. So gcc14/.../gcc13.2 means gcc13.2 was installed with gcc14

[GiudGiud]

I'm glad the issue is solved!

[chunhuizhao478]

Thanks for the help!