Because we leverage computational techniques, we need software. Below we describe the software we've developed or improved for accomplishing good science.

BIGSTICK is a configuration-interaction shell model code originally developed by Johnson and Ormand, under UNEDF’s SciDAC2 program. The code employs the Lanczos algorithm to solve the Schrödinger equation using an m-scheme operation-number representation of Slater determinants. It can be used to find extremum eigenvalues and eigenvectors of a large matrix, but can also calculate inclusive operators, including total response functions and Green’s functions.

BIGSTICK's development continued under SciDAC3 and is MPI/OpenMP aware and has been used to treat bases of size $10^{11}$. Its on-the-fly Hamiltonian construction alleviates communications and storage problems that typically affect Lanczos algorithms on massively parallel machines. BIGSTICK is also well-suited for implementing large HOBET calculations.

HDF5 is a widely-used and professionally-maintained file format and I/O library. It is commonly installed on large supercomputers, and the format can be understood by many third-party software packages such as python or Mathematica. We integrated HDF5 into the widely-used QDP++ library, a lattice QCD framework. We achieved a 30% performance improvement over the commonly-used QIO library.

hypre is a linear solver library that has many advanced techniques for solving linear systems. hypre previously was restricted to at most three dimensions---we generalized it to any number of dimensions. hypre also previously only handled real numbers--we incorporated complex numbers. Both of these improvements are essential for using hypre for lattice QCD. We are interested in applying cutting-edge multigrid methods to lattice QCD problems, as these methods avoid the performance hits that traditional methods suffer from at physically realistic pion mass.

latscat is a measurement code for multi-particle observables in lattice QCD. Built on top of the USQCD software stack, it uses the baryon block formalism and sparse matrix multiplication to efficiently compute the needed Wick contractions. It takes advantage of Fourier acceleration via FFTW for convolutions and uses HDF5 for I/O.

With latscat we performed the first parity-odd two-nucleon scattering calculation from lattice QCD.

METAQ is a small suite of bash scripts that increase the ease and efficiency with which collaborators can use supercomputing resources. Supercomputing time is almost always awarded through a competitive proposal process, and there's usually not enough to make everybody happy. So, if you receive an allocation, you want to be sure you use it as best you can. METAQ helps you avoid wasted cycles, by making it simple to group computational tasks together and to slot smaller and shorter tasks into otherwise-wasted cycles.

METAQ reduced our wasted cycles dramatically, leading to an equivalent of a 25% global software speedup.

mpi_jm is a C++-level resource manager that stresses the shared resources on a supercomputer substantially less than METAQ and allows for much finer control over the high-performance resources. Users' executables are compiled against the mpi_jm library, and tasks are described via a python interface, allowing for easy, high-level description and manipulation. mpi_jm is under developement, and we plan to make a first production-quality version available soon.