PG-PuReMD: properly initialize MPI for threading (MPI_THREAD_FUNNELED). Optimize SAI preconditioner computation to avoid MPI communications when neighbor MPI process is yourself in the presence of periodic boundary conditions.

PG-PuReMD: add OpenMP support for SAI preconditioner computation (--enable-openmp=yes flag in configure script). Other code clean-up.

PG-PuReMD: disable grid optimization which requires the Verlet list be in half format (all MPI+CUDA code and SAI code uses full format). Fix H-bond list initialization issue in MPI-only code. Re-write SAI MPI code to avoid extraneous data movement. Other code clean-up.

PG-PuReMD: simplify preconditioner selection logic. Fix issue with SpMV's applying SAI preconditioning (full row not applied during multiplication). Change solver iterations counts in MPI+CUDA code to match MPI-only code. Other code clean-up.

PG-PuReMD: fixes to SAI preconditioner computation (message size mismatch). Rework MPI communications to not exchange data over MPI when messages are exchanged with oneself (i.e., neighboring processor is oneself via periodic boundary conditions). Backport hydrogen bond fix from sPuReMD (omit interactions when no valid parameters in force field file). Other code clean-up.

PG-PuReMD: standardize and clean-up CUB usage. Add wrapper for non-CUB reductions (rvec, rvec2). Other code clean-up.

PG-PuReMD: add support for using cuBLAS routines for dense linear algebra (mainly level-1 routines in sparse linear solver). Clean-up solver code for using streams.

PG-PuReMD: split van der Waals and Coulomb force and energy computation kernels and execute in seperate streams. Change stream of Coulomb-related kernels (init, charge solver, Coulomb). Use math function for cubic root calculations. Other code clean-up.

PG-PuReMD: default to using bundled CUB library witn recent CUDA SDK version (>= v11.x), while retaining older CUB submodule for older CUDA capabilities (use NVCCFLAGS to manually include CUB in this case). Refactor timing code to more accurately measure kernel timing. Refactor stream logic to expose more parallelism (valence+torsion branch) and to perform fewer synchronizations. Other code clean-up and refactoring.

PG-PuReMD: fix potential MPI message collision issue in SAI preconditioner (same tags to same source rank => use different tags). Reorder communications to allow message size detection and buffer reallocation if needed.

PG-PuReMD: fix bugs in the SAI preconditioner for the charge solver (memory leak, incorrect row numbers).

PG-PuReMD: port SAI preconditioner from MPI codebase. Rework charge solver preconditioner code. Other code clean-up.

PG-PuReMD: fix cross-stream race conditions on intermediary force calculation variables (CdDelta, Cdbo, etc.). Rework valency and torsion calculations to minmize the number of atomic operations performed. Split lone pair and over-/under-coordination kernels. Fix small size mismatch in allocation routines (over-allocation). Other code clean-up.

sPuReMD, PG-PuReMD: add control file keyword (include_polarization_energy) to toggle polarization energy calcuation in ReaxFF.

PG-PuReMD: fix warnings (and errors) from several uninitialized memory regions due to multiple stream change. Fix race condition with valence angle computation due to mistake with dynamic shared memory. Add control file parameter for setting the number of CUDA streams. Other code clean-up.

PG-PuReMD: fix issue with post-processing step with QEq pseudo-charges (too little storage in q array). Fix issue with device-wide CUB reductions and prefix sums reallocating device memory too frequently. Rework CUDA-aware MPI code to fix issues (still seeing out-of-memory errors on Ascent cluster at OLCF).

PG-PuReMD: ensure MPI pack/unpack CUDA kernels are run in the correct stream. Other changes to bring the CUDA-aware MPI functions for the charge solver in line with their MPI counterparts.

PG-PuReMD: utlize multiple CUDA streams to run independent kernels in the ReaxFF+QEq task dependency graph.

PG-PuReMD: fix stream issue with cudaMemset's happening in default stream (switch to cudaMemsetAsync). Rework memory management in light of this change.

WIP: finish the explicit single stream port by using cudaMemcpyAsync and cudaStreamSynchronize calls.

WIP: changes to support multiple CUDA streams. This commit contains porting of all kernels from the default stream to a single non-default stream. This commit has bugs regarding cudaMemcpy's still being in the default stream (and synchronous).

PG-PuReMD: implement MPI+CUDA versions of SDM, PIPECG, PIPECR for charge solver methods (along with missing dual QEq versions).

PG-PuReMD: use a warp of threads for addtional bonded interaction computations (bond orders, bonds, lone pair/over/under).

PG-PuReMD: use a warp of threads for apply bond derivatives to forces and simplify these code paths.

PG-PuReMD: fix incorrect summations in kinetic energy and temperature calculation functions. Replace several hand-coded device-wide reduction kernels with CUB libary reduction.