Running ChemShell: An Advanced Guide

This section outlines the procedures to run ChemShell on parallel computing platforms using MPI. It should be emphasised that due to the variations in operating systems between platform types (and indeed, between different installations running the same machines) that these instructions may need to be modified to reflect local conditions.

External codes

It is possible in principle to launch parallel QM calculations without building or running ChemShell in parallel. Depending on the QM code, use specific environment variables to invoke them in parallel. In some cases, parallel keywords should be specified as an argument to the QM codes. Consult the manual page of the relevant code for more details.

This approach generally works well only on workstations or locally-managed clusters. Other parallel environments, particularly HPC systems, may not allow parallel binaries to be launched from another binary (e.g. if mpirun is not available on the compute nodes). In this case, you must use a code that can be directly linked into a parallel build of ChemShell (see below).

Parallel ChemShell

ChemShell can be executed in parallel if it has been compiled using the --mpi option (see the installation guide). The advantages of running ChemShell in parallel include execution of some internal modules such as DL-FIND in parallel, and the possibility to directly link external codes (see below).

The chemsh script contains the mpirun (or equivalent) command to launch the ChemShell binary (chemsh.x) in parallel. Do not pass the chemsh script as an argument to mpirun as this will result in an error. The -np argument is used to specify the number of processes required. E.g.:

chemsh -np 4 input.py > output.log

will run ChemShell with 4 parallel processes.

As parallel execution is very platform-dependent, you may need to modify the chemsh script for your platform, particularly if mpirun is not used.

For some platforms (e.g. the UK national supercomputer service ARCHER), job submission functionality is included in the script, which can be specified by e.g. --platform=archer and associated arguments.

Alternatively, there may be cases where it is more convenient to call the ChemShell binary directly. In this case, the environment variables that are normally set by the chemsh script must be set manually.

Direct linking of external codes

Some external codes (such as GAMESS-UK and GULP) can be linked in directly as libraries to ChemShell so that the package may be executed in parallel as a single binary. This is more efficient than launching the external codes separately, and is essential on platforms where it is not possible to launch one parallel binary from another (e.g. most HPC facilities do not allow this).

The supported external codes must usually be compiled using special flags to produce a library. Once this is done, direct linking must be specified when ChemShell is compiled using arguments such as --gamess-uk, etc. For further information, see the installation guide.

Once compiled, ChemShell is executed in parallel as described above. If the external code has been directly-lined, the directly-linked library will be automatically used rather than a separate executable.

Task-farming parallelism

MPI parallel builds of ChemShell can run in a task-farm parallel mode. The standard parallel mode is a master/slave configuration in which a Tcl interpreter is run on the master node and the slave nodes remain idle until a parallel input command is executed. All nodes are grouped together in the default MPI World communicator. In the task-farming parallel mode the MPI World communicator is split to form smaller sets of processors called workgroups. The number of workgroups is fixed for the lifetime of the ChemShell run and is specified using the command-line argument -nwg, which is passed to ChemShell when it is executed. For example, the following command would run a 64-processor job using 4 workgroups of 16 processors each:

chemsh -np 64 -nwg 4 input.py > output.log

In each workgroup one node acts as a master and the rest are slaves. Each master runs a copy of the Python interpreter and independently executes the same ChemShell input script.

Global Arrays (GA) memory

NWChem is driven by the GA library. The maximum size of global memory per single process (MPI task) NWChem can allocate is defined as: total memory = heap memory + stack memory + global memory, where heap = stack = global/2. If NWChem is to be called as a directly linked library the GA memory will be managed by ChemShell itself and can only be controlled via command line -gm or --global-memory. For example, the following command allows a maximum global memory of 1024 MB (total 2048 MB):

chemsh -gm 1024 input.py > output.log