1. Ansys CFX Windows Parallel Setup

The following topics are discussed:

 

1.1. Requirements

  • Ansys CFX-Solver must be installed on all hosts that are to be used for a parallel run, or in a location accessible to all hosts.

  • Local parallel runs use Intel MPI (default) for communication. For local parallel operation, the required MPI components are included as part of the Ansys CFX installation and no additional MPI installation or configuration are required.

  • To run distributed parallel (where follower processes run on hosts other than the host with the leader process), the desired MPI service must be installed, configured, and started on all involved hosts. In addition, you must make host information available to the leader host. These tasks are discussed in Setting Up Intel MPI for Windows.

 

1.2. Host Specification

For distributed parallel operation, hosts might need to be predefined in the hostinfo.ccl configuration file. See Ansys CFX Hosts Specification for details.

 

1.3. Setting Up Intel MPI for Windows

To install, configure, and start the Intel MPI service on a given host, install Intel MPI using the Ansys installation launcher. After installing Intel MPI in this way, the MPI service is automatically configured and started. For more information, see Product Installation with Client Licensing.

 

1.3.1. Enabling Intel MPI Parallel Through a Windows Firewall

To enable Intel MPI parallel processing through a Windows firewall:

  1. Click Start > Windows System > Control Panel.

  2. From the Control Panel dialog box, open Windows Defender Firewall.

    If you do not see Windows Defender Firewall listed, click System and Security then open Windows Defender Firewall.

  3. Allow the following programs through the firewall:

    • <Ansys Inc installation path>\v242\CFX\bin\<CFX5OS>\solver-mpi.exe

    • <Ansys Inc installation path>\v242\CFX\bin\<CFX5OS>\double\solver-mpi.exe

    • <Ansys Inc installation path>\v242\CFX\tools\multiport\mpi\win64\intel\bin\mpiexec.exe

    • %I_MPI_ROOT%\intel64\bin\hydra_service.exe

 

1.4. Setting up and Running Microsoft Windows HPC 2019

To set up running Microsoft Windows HPC 2019, steps must be taken so that:

  • The installed software is on a shared location that can be accessed by all hosts.

  • The working directory is a shared location.

  • A mechanism is provided so that all local paths can be properly resolved.

To prepare for running Ansys CFX with Microsoft Windows HPC 2019, you can follow the steps below (terms in angle brackets < > should be replaced with the required entry):

  1. Install Ansys Licensing, Ansys Workbench and Ansys CFX on the head node.

  2. Share the installation directory.

    For example, on a typical installation share C:\Program Files\ANSYS Inc as \\<HeadNodeName>\Ansys Inc where <HeadNodeName> is the name of the head node.

  3. Install the Ansys prerequisites on all of the cluster nodes.

    You can do this either by:

    • Executing the following commands directly on each node:

      \\<HeadNodeName>\ANSYS Inc\v242\prereq\2019vcredist_x64.exe /quiet
\\<HeadNodeName>\ANSYS Inc\v242\prereq\2019vcredist_x86.exe /quiet
\\<HeadNodeName>\ANSYS Inc\v242\prereq\ndp472-kb4054530-x86-x64-allos-enu.exe /q
\\<HeadNodeName>\ANSYS Inc\v242\prereq\ndp48-x86-x64-allos-enu.exe /q

    • or by using the clusrun command on the head node to execute these commands on all the nodes (refer to your Microsoft Windows HPC documentation for details).

  4. Share the working directory of the head node.

    For example, share C:\Users\<UserName> as \\<HeadNodeName>\<UserName> where <UserName> is the username.

    Alternatively, share the working directory of the submitting machine.

    For example, share C:\Users\<UserName> on the submitting machine as \\<SubmitHostName>\<UserName> where <SubmitHostName> is the name of the submitting machine.

  5. If the submitting machine is different from the server head node, it is necessary to install the client utilities of ‘HPC Pack 2019’ on the submitting machine. This is available for download from Microsoft. This sets the CCP_HOME environment variable, enabling the ‘Submit to Windows HPC Queue’ start method in the CFX-Solver Manager.

    To ensure that the client installation includes the necessary software components, the installation should be carried out using the full installation (.zip file rather than .msi file) and choosing the option "Install only the client utilities" from the HPC Pack 2019 installer.

  6. On the submitting machine, create %USERPROFILE%\.cfx\cfxccs_options.txt with the following content to define the required job submission options:

    PATHMAP=C:\Program Files\ANSYS Inc;\\<HeadNodeName>\ANSYS Inc
    PATHMAP=C:\Users\<UserName>;\\<HeadNodeName>\<UserName> or PATHMAP=C:\Users\<UserName>;\\<SubmitHostName>\<UserName> if the working directory has been shared from the submitting machine.
    CLUSTERHOST=<HeadNodeName> to be used when submitting jobs from machines other than the head node.
    PROPAGATE_PARAMETRIC_LICENSE=1 an optional setting to enable parametric licensing. Note that in this case, the submitting machine must have appropriate firewall settings to allow the cluster node(s) to access the license. Typically, this is achieved by adding the executable C:\Program Files\ANSYS Inc\v231\licensingclient\winx64\ansyscl.exe to the list of firewall exceptions on the submitting machine.
    ACCOUNT=<OtherUserDomain>\<OtherUserName> to be used when submitting jobs using different credentials, where <OtherUserDomain> and <OtherUserName> are the domain and username of another user, respectively.
    PROCESSORSPERSOLVER=2 an optional setting (default setting is 1) that allocates the number of cores per partition. This is typically used on hosts that are limited by memory bandwidth such as Xeon-based machines.

  7. Set up Ansys Workbench for a network as described in Network Installation and Product Configuration.

To submit a job:

  1. Start the CFX-Solver Manager (either using CFX standalone or from Ansys Workbench) on the submitting machine using the software installed on the head node.

  2. Ensure that the Run Mode on the Run Definition tab of the CFX-Solver Manager is set to Submit to Windows HPC Queue. Otherwise, set up the CFX-Solver Manager as normal.

  3. Click Start Run on the CFX-Solver Manager to start the simulation. The first time a simulation is run, supply the required credentials that are prompted (this prompt may appear behind the CFX-Solver Manager window). You may elect to have these credentials saved to avoid repeated prompting of these credentials.

Note:

  1. %USERPROFILE% is typically:

    C:\Users\<UserName>

  2. For Ansys Workbench with Parameters and Design Points, start the CFX-Solver Manager, ensure that the Run Mode on the Run Definition tab of the CFX-Solver Manager is set to Submit to Windows HPC Queue, set the maximum number of processes, then click Save Settings. When the user clicks Update all Design Points, each parameter or design point will be solved on the cluster.