The following topics are discussed:
The Ansys CFX-Solver must be installed on all hosts that are to be used for a parallel run, or in a location accessible to all hosts.
To run distributed parallel (where follower processes run on a different host from the leader process), remote or secure shell access must be available from the leader nodes (systems on which parallel runs will be started) to follower nodes (systems on which the Ansys CFX-Solver will actually run). See Setting Up Remote Access on Linux, below.
You must have the same username on all systems.
For distributed parallel operation, hosts might need to be predefined in the hostinfo.ccl configuration file. See Ansys CFX Hosts Specification for details.
Each system that will be used as a follower node must be configured to allow access via remote shell commands from the leader node. This can be done globally for all users or on a per-user basis.
Often, networks where ssh, rsh or rlogin are used frequently will already be configured to allow remote access to all users. If this is the case, nothing more need be done.
The CFX5RSH environment variable is used to select the
remote shell protocol. By default, CFX5RSH=ssh is set. If
you want to use rsh, then set CFX5RSH=rsh either in the
environment or in the cfx5rc file (see Resources Set in cfx5rc Files in the CFX Introduction).
In either case, ssh or rsh, the remote machine must not prompt for a password when you run remote commands.
You can test remote access using ssh using the command:
ssh unixhost echo working
You can test remote access using rsh for a Linux follower node using the command:
rsh unixhost echo working
If you use ssh for remote access, consult your system documentation on how to set up ssh between machines so that it does not require the user to enter a password.
This method, which is used to allow remote access for all users, depends on whether NIS is used to maintain netgroups, as well as common password and group databases. If this is not the case, then you should log in to each follower node as root and create a file called /etc/hosts.equiv containing a line:
<leader>where <leader> is the hostname of the leader
node. See your system documentation for more information about the use of rsh
and the syntax of the hosts.equiv file.
If NIS is used to control remote access, then a netgroup must be created for users of Ansys CFX by the root user on the NIS leader server, and a line such as the one below added to /etc/hosts.equiv on each follower node by the root user:
+@<netgroup>where <netgroup> is the name of a netgroup to
which users of Ansys CFX belong. A detailed description of how to configure NIS
is beyond the scope of this manual. See your system documentation for more
information about NIS.
An individual user can create a file called .rhosts in their home directory on each follower containing a line:
<leader> <user>
where <leader> is the hostname of the leader and
<user> is the user’s username on the leader. This
file should be made readable only by the user, for example, by running:
chmod 600 ~/.rhosts
Most Linux systems support the Intel MPI run mode. Intel MPI is the preferred parallel run mode on most of the supported Ansys CFX platforms.
Intel MPI supports several network fabrics including:
Shared memory, typically used for intra-host communication.
DAPL-capable network fabrics, such as Infiniband, iWarp, Dolphin and XPMEM (through DAPL).
TCP/IP-capable network fabrics, such as Ethernet and InfiniBand (through IPoIB).
TMI-capable network fabrics including Qlogic and Myrianet (through Tag Matching Interface).
OFA-capable network fabrics including InfiniBand (through OFED verbs).
Intel MPI is automatically installed with Ansys CFX in the <CFXROOT>/../commonfiles/ directory. You do not need a license from Intel to run the Ansys CFX-Solver using Intel MPI.
Intel MPI run modes are selected in the Ansys CFX-Solver Manager or on the command line (using the
-start-method command line option) using the "Intel MPI Local Parallel"
or "Intel MPI Distributed Parallel" start method options.
In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.
Ansys CFX uses the environment variable CFX5_IMPI_DIR to select which Intel MPI
installation is used. The default setting points to the version that is
automatically installed by Ansys CFX in
the <CFXROOT>/../commonfiles/MPI/Intel directory. You
can use cfx5info -v to find the current settings for CFX5_IMPI_DIR and other
environment variables. If you want to use a different Intel MPI version, you can
install that version in an alternative location and set CFX5_IMPI_DIR to that
location.
There are several environment variables that can be used to control Intel MPI that may provide additional flexibility that is not directly accessible through the Ansys CFX start-up scripts. For example, environment variables are available for network fabric selection (discussed briefly below), process pinning control, and collective operation control.
Any of these environment variables can be set by modifying the start command
in the etc/start-methods.ccl file and using the Intel MPI
options -env and/or -genv (an example of this usage is given in the section on
network fabric selection). More detail on all options and environment variables
is available in the Intel MPI documentation.
Some useful environment variables include those described below. If these environment variables are set, the start-up scripts automatically set them for your Ansys CFX runs.
- I_MPI_MPD_REMSH
Specifies which method is used by Intel MPI to spawn jobs on remote machines. The Intel MPI default is ssh. However, by default the Ansys CFX software sets this to be the same as the setting used for
CFX5RSH, and rsh ifCFX5RSHis not set. This variable is applicable only to Linux.- I_MPI_FABRICS and I_MPI_FABRICS_LIST
Specifies the network fabrics to be used and the order in which Intel MPI searches for them.
- I_MPI_HYDRA_IFACE
Specifies the network interface, for example "eth0".
Intel MPI start methods use the default network fabric selection built into
Intel MPI. The default order in which Intel MPI checks for fabrics is DAPL, OFA,
TCP, TMI. The environment variable I_MPI_FABRICS_LIST controls the search order
used by Intel MPI.
If you want to force selection of a specific network fabric, then you can
either set the I_MPI_FABRICS environment variable or modify the start command in
the etc/start-methods.ccl file and add the necessary
command line flags (for example: -env IMPI_FABRICS shm:dapl to select the
combination of shared memory and DAPL fabrics).
The following table shows how Intel MPI searches for drivers for the various network fabrics:
| Interconnect | 1st attempt | 2nd attempt |
|---|---|---|
| shm | N/A | N/A |
| DAPL | Environment variable I_MPI_DAT_LIBRARY | libdat.so or libdat.so.1 for DAPL 1.2 providers and libdat2.so or libdat2.so.1 for DAPL 1.2 providers |
| TCP | N/A | N/A |
| TMI | Environment variable I_MPI_TMI_LIBRARY | libtmi.so |
| OFA | Environment variable I_MPI_OFA_LIBRARY | libibverbs.so |
For example, Intel MPI expects that the DAT shared library for the DAPL
network fabric is located either in the default library search path or in a
specific location if a full path is specified. If the shared library cannot be
found, Intel MPI will not take advantage of the DAPL network fabric. To fix
this, you will have to install the shared libraries into the correct location or
set the environment variable I_MPI_DAT_LIBRARY to point to the correct DAT
shared library.
There are other environment variables not documented here that provide further control over network fabric selection. These are described in the Intel MPI documentation.
The Ansys CFX-Solver supports Cray’s native implementation of MPI on the Cray Linux Environment (CLE) using the start method ‘Cray MPI Distributed Parallel’.
Cray MPI is closely integrated with the batch queuing system and interconnects used on Cray systems.
Cray MPI is the recommended method of running on Cray systems, as opposed to using Cluster Compatibility Mode (CCM) with a different MPI.
When starting a solver run, the Cray MPI start method may be specified on the
command line using the option: -start-method “Cray MPI
Distributed Parallel”, having reserved the required
processing elements via the built-in batch queuing
system.
The aprun command, which can be modified, is used by
the start method to launch the job. Other options available from Cray are
described in the documentation for aprun.
Note that numa node memory containment is enforced, using the
-ss option. The default method used by Cray MPI to
allocate processes to CPU cores is appropriate when all cores are used. However,
if it is intended that each host is only partly utilized (for example, because
of the memory requirements of the run), this can result in asymmetric
utilization. In such cases, the -S option can be used to
limit the number of processes per numa node. For example, -S
4 would limit the number of processes per numa node to 4.