31.3.1. Configuring a Distributed-Memory Parallel (DMP) Analysis

This section reviews the required configuration for using Ansys Polyflow via MPI on Windows and Linux.

There are some limitations for the DMP Analysis with Ansys Polyflow.

  • Only Intel MPI is supported.

  • The Polyflow leader must be launched from the folder containing the simulation files.

  • If the Polyflow leader is launched in a remote folder with a UNC path, system functions called by Polyflow can generate the message: 

    '<UNC_path_to_the_remote_folder>'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported. Defaulting to Windows directory.

    This message is generated by the system and has no impact on the run.

In this section, it is assumed that Ansys products and Intel MPI are installed. Refer to Configuring a Distributed-Memory Parallel (DMP) Analysis and Configuring High Performance Computing Guide in the Ansys, Inc. Installation Guides for more details.

 

31.3.1.1. DMP Analysis Under Windows

This section shows how to launch a DMP Analysis with polyflow_mpi.

machine_I is the machine where the Ansys products are installed.

machine_W is the machine where the working directory resides.

Below are the three possible ways for all nodes to access executables.

Installation of Ansys Products on All Machines at the Same Location C:\Program Files\Ansys Inc

  • Install Ansys products on all machines in C:\Program Files\ANSYS Inc.

  • The command to launch polyflow_mpi is: 

    C:\Program Files\ANSYS Inc\v242\polyflow\ntbin\win64\polyflow_mpi

Installation of Ansys Products on One Machine Using UNC Paths to Launch polyflow_mpi

  • Install Ansys products on machine_I in C:\Program Files\ANSYS Inc. and share this folder.

    For example, C:\Program Files\ANSYS Inc shared as \\machine_I\Ansys Inc.

  • The command to launch polyflow_mpi is:

    \\machine_I\ANSYS Inc\v242\polyflow\ntbin\win64\polyflow_mpi

Installation of Ansys Product on One Machine and Shared via Mapped Network Drive

  • Install Ansys products on machine_I in C:\Program Files\ANSYS Inc.

  • On machine_I, share the installation folder C:\Program Files\ANSYS Inc.

    • Specify the <shared name>, for example, Ansys_Inc.

    • Be sure Read and Execute permissions are set for all users which will use Ansys Polyflow.

    • Other machines will view the shared folder as \\machine_I\<shared name>.

  • For example, on machine_I, define a Mapped network drive for the shared installation directory and use drive letter I:.

  • On each client machine, define the same Mapped network drive to the shared installation directory of the machine_I.

  • The command to launch polyflow_mpi is: 

    I:\v242\polyflow\ntbin\win64\polyflow_mpi

Recommendation to Installing and Launching polyflow_mpi

Assuming that Intel MPI is installed on all machines, the simplest way to run polyflow_mpi on multiple machines while minimizing machine configuration is as follows:

  1. Install Ansys products on Machine_I.

  2. On Machine_I, share the installation folder C:\Program Files\ANSYS Inc as \\machine_I\Ansys Inc.

  3. Machine_L is the machine running the Polyflow leader.

  4. Machine_F are the machines running the Polyflow followers.

  5. The working directory must be accessible by Machine_L via the UNC path : <UNC_path_to_working_directory>.

  6. To run a simulation from the working directory <UNC_path_to_working_directory> on <hostname1> and <hostname2>, type the following command:

    “\\machine_I\Ansys Inc \v242\polyflow\ntbin\win64\polyflow_mpi” -mpi <# of processes> -ppn <# of processes per node> 
-hosts < Machine_L >,< hostname1>,< hostname2>, -keyword FORCE_SOLVER=MUMPS -ID <datafile> -OUT <listingfile>

If the working directory is not on Machine_L, add -wDir <UNC path to working directory>.

  • -mpi <# of processes>

    Total number of MPI threads.

  • -ppn <# of processes per node>

    Number of MPI threads per machine/node.

  • -hosts <hostname1>,<hostname2>

    List of hostnames.

  • -wdir :W:\<sim1>

    Shared simulation folder.

  • -th <# of processes per thread>

    Number of OMP processes per MPI thread used by BLAS functions in MUMPS. It also represents the number of processes used to build the matrix in parallel on the leader machine.

  • -keyword FORCE_SOLVER=MUMPS

    Required if the MUMPS solver has not been selected in the setup during the polydata session.

Recommendation for Number of Processes per Nodes (-ppn).

You do not need to have the number of processes per node (-ppn) set to greater than one for the memory footprint reduction of DMP analyses.

Other Polyflow arguments, including -ID, -OUT and -s remain valid for DMP analysis.

Other MPI arguments include:

  • -cnf <hostfile>

    Text file listing the machine names (one machine per line). This replaces the -hosts argument.

  • -mapall

    Maps all network drives. If the command uses the map network drive I: or W:, this parameter is used.

  • -verbose 

    Enters verbose mode.

  • -print-rank-map

    Prints rank mapping at the beginning.

Polyflow Leader and Followers

The machine/node appearing first in the -hosts or cnf_hostfile argument will be the Polyflow leader and others will be the Polyflow follower(s).

If a problem occurs with MPI, you can add the -verbose argument to the polyflow_mpi command. This prints relevant information related to the issue.

Location of Files for Buffering on Disk

Aside for the buffering on disk, the followers do not read or write files. If buffering on disk is active, MUMPS will create folders and files in the directory specified by the TMP environment variable.

Note:  This folder must exist on all nodes, therefore leader and followers. The path contained in the TMP environment variable must be valid on all nodes.

 

31.3.1.2. DMP Analysis Under Linux

This section explains how to configure your Linux network/cluster to run a DMP Analysis with Ansys Polyflow. Refer to Configuring a Distributed-Memory Parallel (DMP) Analysis and Configuring High Performance Computing Guide in the Ansys, Inc. Installation Guides for additional information.

Requirements

  • Ansys Polyflow must be installed on all hosts used for a parallel run or in a location accessible to all hosts.

  • To run distributed parallel, where the follower processes the run on a different host from the leader process, secure shell access must be available from the leader nodes (systems on which parallel runs will be started) to follower nodes (systems on which Ansys Polyflow will actually run). See Setting up Remote Access on Linux below.

  • You must have the same username on all systems.

Installing the Software

Install Ansys 2024 R2 following the instructions in the Configuring High Performance Computing Guide in the Ansys, Inc. Installation Guides for your platform. Be sure to complete the installation, including all required post-installation procedures.

To run on a cluster, you must:

  • Install Ansys 2024 R2 on every machine located on the cluster. Each machine must have Ansys 2024 R2 in the same location.

  • You can use the exported NFS file systems on Linux. Install Ansys 2024 R2 on one Linux machine (for example, at /ansys_inc/v242). You will then export this directory. On the other cluster machines, create an NFS mount from the first machine to the same local directory (/ansys_inc/v242).

Setting up Remote Access on Linux

Each system used as a follower node must be configured to allow access via remote shell commands from the leader node. This can be done globally for all users or on a per-user basis.

Usually, networks with ssh frequently used will already be configured to allow remote access to all users. In these cases, nothing more needs to be done.

Most importantly, the remote machine must not prompt for a password when you run remote commands.

Testing Remote Access

You can test remote access using ssh with the following command:

ssh unixhost echo working

Consult your system documentation on how to set up ssh between machines so that it does not require you to enter a password.

Using Intel MPI (Message Passing Interface Library)

Most Linux systems support the Intel MPI run mode. Intel MPI is the preferred parallel run mode for Ansys Polyflow.

Intel MPI supports several network fabrics including:

  • Shared memory, typically used for intra-host communication.

  • DAPL-capable network fabrics, such as Infiniband, iWarp, Dolphin and XPMEM (through DAPL).

  • TCP/IP-capable network fabrics, such as Ethernet and InfiniBand (through IPoIB).

  • TMI-capable network fabrics including Qlogic and Myrianet (through Tag Matching Interface).

  • OFA-capable network fabrics including InfiniBand (through OFED verbs).

Intel MPI is automatically installed with Ansys Polyflow in the .../Ansys_inc/v242/commonfiles/ directory.

You do not need an Intel license to run Ansys Polyflow using Intel MPI.

Environment Variables

In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.

There are several environment variables that can be used to control Intel MPI that can provide additional flexibility not directly accessible through Ansys Polyflow start-up scripts. For example, environment variables are available for network interface selection, process pinning control and collective operation control. More details on options and environment variables are available in the Intel MPI documentation https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library-documentation.html.

Some useful environment variables include those described below. If these environment variables are set, the start-up scripts automatically take them into account for your Ansys Polyflow runs.

I_MPI_HYDRA_IFACE

Specifies the network interface, for example eth0.

I_MPI_HYDRA_DEBUG

Prints out debug information to help troubleshoot MPI issues.

Launching the Polyflow Solver in Distributed Memory

To launch Ansys Polyflow on several machines, type the following command:

/ansys_inc/v242/polyflow/bin/polyflow_mpi -mpi <# of processes>
 -ppn <# of processes per node> -hosts <hostname1>,<hostname2>
 -th <# of OMP processes per thread> -keyword FORCE_SOLVER=MUMPS -ID
 <datafile> -OUT <listingfile)

  • -mpi <# of processes>

    Total number of MPI threads.

  • -ppn <# of processes per node>

    Number of MPI threads per machine/node.

  • -hosts <hostname1>,<hostname2>

    List of hostnames.

  • -th <# of processes per thread>

    Number of OMP processes per MPI thread used by BLAS functions in MUMPS. It is also the number of processes used to build the matrix in parallel on the leader machine

  • -keyword FORCE_SOLVER=MUMPS

    Required if the MUMPS solver has not been selected in the setup during the polydata session.

You do not need to have the number of processes per node (-ppn) set to greater than one for the memory footprint reduction of DMP analyses.

Other Polyflow arguments, including -ID, -OUT and -s remain valid for DMP analysis.

Other MPI arguments include:

  • -cnf <hostfile>

    Text file listing the machine names (one machine per line). This replaces the -hosts argument.

  • -verbose 

    Enters verbose mode.

  • -print-rank-map

    Prints rank mapping at the beginning.

Polyflow Leader and Followers

The machine/node appearing first in the -hosts or cnf_hostfile argument will be the Polyflow leader and others will be the Polyflow follower(s).

If a problem occurs with MPI, you can add the -verbose argument to the polyflow_mpi command. This prints relevant information related to the issue.

Location of Files for Buffering on Disk.

Aside for the buffering on disk, the followers do not read or write files. If buffering on disk is active, MUMPS will create folders and files in the directory specified by the TMPDIR environment variable.

Note:  This folder must exist on all nodes, therefore leader and followers. The path contained in the TMPDIR environment variable must be valid on all nodes.