Installation Guide

A guide to build EXESS for the HPC systems expert that will be building it.

EXESS has a minimal set of dependencies and our strong code writing policies have caught most compiler errors that might arise when going from gcc to xlc, to nvhpc, to cray-cc etc. However, we can’t guarantee that the software will compile out of the box for non-standard compilers or those our team does not have access to for testing.

The main dependencies are:

  • A C/C++ compiler with at least C++17 capabilities

  • A CUDA/ROCM compiler. CUDA and ROCM versions will be detailed below

  • A functioning MPI library (versions and flavors will be discussed below)

  • OpenMP support

  • An HDF5 install

  • For AMD systems a Magma installation with HIP support

Listed here are the steps to build the code on the Gadi HPC system at the NCI in Canberra, and the Setonix HPC system at the Pawsey Supercomputing Centre in Perth.

Gadi @NCI

module load julia/1.9.1
module load cuda/12.0.0
module load openmpi/4.0.1
module load hdf5/1.12.1
module load gcc/12.2.0
module load cmake/3.24.2
module load intel-mkl/2023.2.0
module load python3/3.10.0

mkdir build
cd build
CUDAARCHS="70;80" cmake -DCMAKE_INSTALL_PREFIX=$PATH_TO_INSTALL ../
make -j install

Setonix @Pawsey

Installing on Setonix is more involved due to the need of Magma. We assume a module for Magma has been created by the staff and is readily available for EXESS.

module load gcc/12.2.0
module load cray-hdf5/1.12.2.7
module load rocm/5.7.3
module load cmake/3.27.7
module load magma/2.8.0-${custom}
module load craype-accel-amd-gfx90a
module load julia
export MPI_ROOT=$MPICH_DIR
export MPICH_GPU_SUPPORT_ENABLED=1

mkdir build
cd build
cmake .. -DGPU_RUNTIME=HIP -DMPI_ROOT=$MPI_ROOT -DCMAKE_HIP_ARCHITECTURES=gfx90a -DCMAKE_INSTALL_PREFIX=$PATH_TO_INSTALL ../
make -j install

Necessary environment variables

EXESS simply needs to know where the files it needs are, these are basically the basis sets and guesses in the records path. The validation path is only used for running the Julia validation scripts. The EXESS path will point at the ${PROJECT_SOURCE_DIR} CMake variable, since it will assume that the records/ and validation/ directories are there.

After EXESS has been installed, the single thing needed by the program from the source tree is the EXESS_RECORDS_PATH, the run.sh script and the runexess script; anything else can be safely deleted. Below is the automatically generated file set_exess_vars.sh which contains the env variables neede by EXESS.

export EXESS_PATH="$YOUR_PATH/exess"
export EXESS_RECORDS_PATH=$EXESS_PATH/records
export EXESS_VALIDATION_PATH=$EXESS_PATH/validation

Custom installers

In the modulefiles/ directory a custom install script for Gadi and Setonix is available. These files will create and install a module for EXESS. Feel free to use these as a base for however you’d like to configure your build.

A note on programming environments

GNU has proven to be by far the best environment to build EXESS in. The code will build with the Cray compiler suite, the NVHPC compilers, and the Intel compilers. Please report any errors arising during compilation for any type of compiler, math library, MPI, and cuda/hip version to the EXESS team.

We are working on providing a full list of compiling environments in both Gadi and Setonix that function and perform at the highest EXESS standards.