Extending Scheduler Integration with Tight Integration for MPI
AnsysEM supports tight integration with a few popular schedulers by leveraging built-in tight-integration capabilities of Intel MPI. Specifically on Linux, this works out of the box with both LSF and GE. For other Linux based schedulers, this note describes how scheduler integration developers can extend scheduler integration to provide tight integration for MPI based solves.
MPI Based Solves
Like many other simulation applications, AnsysEM also uses an MPI-based message passing approach to provide portable parallel execution on shared, distributed and distributed shared memory computers.
An MPI solve involves running a mpiexec (or mpirun) command that runs the simulation’s executable on each allocated/specified host. For most applications, the entire simulation is just a single MPI solve. However, an AnsysEM simulation consists of multiple MPI solves which are organized as following:
- Multiple MPI solves in series: Each AnsysEM simulation runs multiple MPI solves.
- Different host list: Each such MPI solves may use different amount of resources (mostly cores per host) that is specified using the host list and count of ranks.
- Multiple MPI solves in parallel: In some case, multiple MPI solves may be launched in parallel from different hosts. So multiple MPI solves may be running simultaneously.
- Multiprocessing: Finally, within each MPI based solve, each rank can use multiprocessing as well.
To launch an MPI based solve AnsysEM code uses an MPI integration-related wrapper file, called mpirun.fl, which in turn invokes MPI’s mpirun/mpiexec. Ansys EM installation includes all the needed files for the supported MPI implementations.
Scheduler integration for MPI
When a scheduler runs a simulation, each scheduler job usually runs just a single MPI solve. Each time MPI needs to launch a process on a remote allocated host, it will do so either with password-less ssh (loose integration), or with a scheduler specific script/command (tight integration).
Custom tight-integration
For custom tight integration, you provide a script that uses scheduler's utilities to launch processes on a remote host. The script needs to be a drop-in replacement for ssh. For AnsysEM to use this script for MPI solves, you need to specify the following environment variables and provide a bootstrap selector and a bootstrap script file.
| Environment Variables | Value | Description |
|---|---|---|
| ANSYSEM_HPC_MPI_USE_SCHEDULER_CMD | A keyword that identifies the scheduler | Scheduler for which to enable tight integration |
| ANSYSEM_HPC_MPI_CUSTOM_SCRIPT_FILE | Fully qualified path to the custom script file | Script that implements a function that returns the switch to be used for MPI implementation’s command line. |
Script Files
| Script Files | Content | Description |
|---|---|---|
| Bootstrap selector BASH script file | implements a bash function that returns the switch to be used for MPI implementation’s command line. |
Intel MPI: Implements ansysem_hpc_intel_mpi_bootstrap either returns |
| Bootstrap BASH script file | Shell script that is a ssh drop-in replacement | Launches a process on a remote host. Accepts the same arguments as ssh |
Example: PBSPro
For custom tight integration with PBSPro, you provide a script that uses the pbsdsh utility to launch processes on a remote host. The script is a drop-in replacement for ssh. It assumes that the first two arguments are -x and -q and it provides a bootstrap selector and bootstrap script file.
For an AnsysEM batch solve running under PBSPro to use this script for MPI solves, the job script will set the environment variables that are listed in the examples that follow.
Job script excerpt
# The selector script will specify the MPI options for PBSPro
export ANSYSEM_HPC_MPI_USE_SCHEDULER_CMD=pbspro
# Bootstrap selector BASH script file
export ANSYSEM_HPC_MPI_CUSTOM_SCRIPT_FILE=${ANSYSEM_DIR}/v<version>/Linux64/schedulers/utils/pbspro-intelMPI-integration.sh
# Special environment variable to enable PBS Node file to become available in Bootstrap script file.
export ANSYSEM_HPC_MPI_SCHEDULER_NODEFILE=$PBS_NODEFILE
pbspro-intelMPI-integration.sh
|
#!/bin/bash function ansysem_hpc_intel_mpi_bootstrap() { local custom_scheduler="$1" local bootstrap_switch="" if [[ -n "$custom_scheduler" ]] then if [[ "$custom_scheduler" == "pbspro" ]] then exec_file=$(dirname $ANSYSEM_HPC_MPI_CUSTOM_SCRIPT_FILE) exec_file_name=pbspro_dsh_wrapper.sh bootstrap_switch="-bootstrap-exec $exec_file/$exec_file_name" fi fi echo "$bootstrap_switch" } |
pbspro_dsh_wrapper.sh
|
#!/bin/bash # Bootstrap BASH script file for PBSPro # Assumes that the first two arguments are -x and -q # For debugging, set it to a suitable path # DEBUG_DIR=$(HOME)/Ansoft/Test/DebugLog PBS_BINDIR=/opt/pbs/bin shift 2 host_name=$1 # Assumes that the next argument is host name shift 1 # Compute the index for the first entry index1=$(( $(grep -n -m 1 $host_name $ANSYSEM_HPC_MPI_SCHEDULER_NODEFILE | cut -f1 -d:) - 1)) if [[ -n "$DEBUG_DIR" ]] then echo "running pbsdsh ON $host_name as index $index1 $@" >> ${DEBUG_DIR}/wrapper.log cat $ANSYSEM_HPC_MPI_SCHEDULER_NODEFILE >> ${DEBUG_DIR}/wrapper.log fi $PBS_BINDIR/pbsdsh -v -n $index1 -- "$@" |
Intel MPI custom bootstrap feature
From Intel MPI docs:
-bootstrap-exec <bootstrap server>
Use this option to set the executable to be used as a bootstrap server. For example:
> mpiexec -bootstrap-exec <bootstrap_server_executable> -f hostfile -env <VAR1> <VAL1> -n 2 test.exe
This -bootstrap-exec switch lets you specify a shell script that is a ssh drop-in replacement. For custom tight integration, AnsysEM uses Intel MPI and leverages its special switch. This -bootstrap-exec switch lets you specify a shell script that is a ssh drop-in replacement. Using it requires a small patch to an MPI integration related wrapper file, called mpirun.fl. This file, which is available in AnsysEm installation, is used by AnsysEM code to launch an MPI based solve (similar to using mpirun). Each time Intel MPI needs to launch a process on a remote allocated host, it will call a scheduler- (or user-) specified script using arguments that are same as those for password-less ssh.