During MAX, a number of open source materials science domain specific libraries have been developed. The libraries, that are specific to the materials modelling domain, contain functions which perform basic quantum chemistry materials science operations like computation of Hamiltonian eigenstates, charge density and potential, atomic forces, Poisson solvers, large-matrix iterative solvers for eigenvalue problems and linear systems, non-linear optimisation solvers at various levels of generality, symmetry and crystal-structure managements, etc. In parallel, mathematical and system libraries – which perform general-purpose system tasks (such as e.g. memory, communication, and mass-storage management) or mathematical tasks (typically, but not limited to, linear algebra and signal analysis such as FFT or wavelet analysis) have been implemented. Appropriate application program interfaces (API’s) to access domain-specific libraries have been also developed.
CheSS - LAXLib & FFTXLib - Sirius - DBCSR - COSMA - SpFFT
CheSS
One of the most important tasks in electronic structure codes is the calculation of the density matrix. If not handled properly, this task can easily lead to a bottleneck that limits the performance of the code or even renders big calculations prohibitively expensive.
CheSS is a library that was designed with the goal of enabling electronic structure calculations for very big systems. It is capable of working with sparse matrices, which naturally arise if big systems are treated with a localized basis. Therefore, it is possible to calculate the density matrix with O(N), i.e., the computation cost only increases linearly with the system size.
The CheSS solver uses an expansion based on Chebyshev polynomials to calculate matrix functions (such as the density matrix or the inverse of the overlap matrix), thereby exploiting the sparsity of the input and output matrices. It works best for systems with a finite HOMO-LUMO gap and a small spectral width. CheSS exhibits a two-level parallelization using MPI and OpenMP and can scale to many thousands of cores. It has been converted into a stand-alone library starting from the original codebase within BigDFT. At the moment, it is coupled to the two MAX flagship codes BigDFT and SIESTA.
The performance of CheSS has been benchmarked against PEXSI and (Sca)LAPACK for the calculation of the density matrix and the inverse of the overlap matrix, respectively. CheSS is the most efficient method, as it is demonstrated with more details and performance figures in the publication "Efficient Computation of Sparse Matrix Functions for Large-Scale Electronic Structure Calculations: The CheSS Library".
Download
LAXLib & FFTXLib
One of the most important obstacles when keeping the codes up to date with hardware is the programming style based on old-style (i.e., non object-oriented) languages. Programming styles in community codes are often naive and lack the modularity and flexibility. From here, the need to disentangle such codes is essential for implementing new features or simply refactoring the application in order to efficiently run on the new architectures. Rewriting from scratch one of these codes is not an option because the communities behind these codes would be disrupted. One of the possible approaches that could permit to evolve the code is to progressively encapsulate the functions and subroutines, breaking up the main application in small (possibly weakly dependent) parts.
This strategy was followed by Quantum ESPRESSO: two main types of kernels were isolated in the independent directories and proposed as candidates for the domain-specific libraries for third-party applications.
The first library, called LAXlib, contains all the low-level linear algebra routines of Quantum ESPRESSO, and in particular those used by the Davidson solver (e.g., the Cannon algorithm for the matrix-matrix product). The LAXlib also contains a mini-app that permits to evaluate the features of a HPC interconnect measuring the linear algebra routines contained therein.
The second library encapsulates all the FFT related functions, including the drivers for several different architectures. The FFTXlib library is self-contained and can be built without any dependencies on the remaining part of the Quantum ESPRESSO suite. Similarly, in the FFTXlib there is a mini-app that permits to mimic the FFT cycle for the SCF calculation of the charge density tuning the parallelization parameters of Quantum ESPRESSO. This mini-app has also been used to test the new implementation using the MPI-3 non blocking collectives.
Download
Sirius
SIRIUS is a domain specific library for electronic structure calculations. It implements pseudopotential plane wave (PP-PW) and full potential linearized augmented plane wave (FP-LAPW) methods and is designed for GPU acceleration of popular community codes such as Exciting, Elk and Quantum ESPRESSO. SIRIUS is used in production at CSCS to enable QE on GPUs. The library is open-source (BSD 2-clause licence) and is freely available. SIRIUS is written in C++11 with MPI, OpenMP and CUDA/ROCm programming models. SIRIUS is organised as a collection of classes that abstract away the different building blocks of DFT self-consistency cycle.
Download
SIRIUS GitHub
Online documentation
DBCSR
DBCSR is a library designed to efficiently perform sparse matrix-matrix multiplication, among other operations. It is MPI and OpenMP parallel and can exploit Nvidia and AMD GPUs via CUDA and HIP. The library is open-source (GPL v2.0) and is freely available.
Download
COSMA
COSMA is a parallel, high-performance, GPU-accelerated, matrix-matrix multiplication algorithm and library implementation that is communication-optimal for all combinations of matrix dimensions, number of processors and memory sizes, without the need for any parameter tuning. COSMA is written in C++11 with MPI, OpenMP and CUDA/ROCm programming models. The library is open-source (BSD 3-clause licence) and is freely available.
Download
SpFFT
SpFFT is a 3D FFT library for sparse frequency domain data written in C++ with support for MPI, OpenMP, CUDA, and ROCm. It was originally intended for transforms of data with spherical cutoff in frequency domain, as required by some computational materials science codes. For distributed computations, SpFFT uses a slab decomposition in space domain and pencil decomposition in frequency domain (all sparse data within a pencil must be on one rank). The library is open-source (BSD 3-clause licence) and is freely available.
Download
SpFFT GitHub
Online documentation