You can check the latency and bandwidth of the overall cluster interconnect, to help identify any issues affecting Ansys Fluent scalability, by clicking Latency and Bandwidth in the Parallel ribbon tab (Network group box).

 Parallel → Network → Latency

Depending on the number of machines and processors being used, a table containing information about the communication speed for each node will be displayed in the console. The table will also summarize the minimum and maximum latency between two nodes.

Consider the following example when checking for latency:

Latency (usec) with 1000 samples [1.83128 sec]
 ------------------------------------------
 ID       n0     n1     n2    n3    n4    n5
 ------------------------------------------
 n0            48.0   48.2  48.2  48.3   *50 
 n1    48.0           48.2  48.3  48.3   *48 
 n2    48.2    48.2         48.8  49.1   *53 
 n3    48.2    48.3    *49        48.6  48.5 
 n4    48.3    48.3   49.1  48.6         *50 
 n5    49.7    48.5    *53  48.5  49.7      
 
 ------------------------------------------
 Min: 47.9956 [n0<-->n1]
 Max: 52.6836 [n5<-->n2]
 ------------------------------------------

Six processors (n0 to n5) are spawned. The latency between n0 and n1 is 48.0 . Similarly, the latency between n1 and n2 is 48.2 . The minimum latency occurs between n0 and n1 and the maximum latency occurs between n2 and n5, as noted in the table. Checking the latency is particularly useful when you are not seeing expected speedup on a cluster.

 Parallel → Network → Bandwidth

In addition to checking for latency, you can check your bandwidth. A table containing information about the amount of data communicated within one second between two nodes is displayed in the console. The table will also summarize the minimum and maximum bandwidth between two nodes.

Consider the following example when checking for bandwidth:

 Bandwidth (MB/s) with 5 messages of size 4MB [4.36388 sec]
 --------------------------------------------
 ID      n0     n1     n2     n3     n4     n5
 --------------------------------------------
 n0          111.8    *55  111.8   97.5  101.3 
 n1   111.8          69.2   98.7  111.7    *51 
 n2    54.7   69.2          72.9  104.8    *45 
 n3   111.8   98.7   72.9          64.0    *45 
 n4    97.6  111.7  104.8    *64          76.9 
 n5   101.2   50.9   45.5    *45    76.9    
 
 --------------------------------------------
 Min: 45.1039 [n5<-->n3]
 Max: 111.847 [n0<-->n3]
 --------------------------------------------

The bandwidth between n0 and n1 is 111.8 MB/s. Similarly, the bandwidth between n1 and n2 is 69.2 MB/s. The minimum amount of bandwidth occurs between n3 and n5 and the maximum occurs between n0 and n3, as noted in the table. Checking the bandwidth is particularly useful when you cannot see good scalability with relatively large cases.

In Ansys Fluent, you can reduce communication and improve parallel performance by increasing the report interval for residual printing/plotting or other solution monitoring reports. You can modify the value for Reporting Interval in the Run Calculation Task Page.

 Solution → Run Calculation Calculate...

When using the Surface to Surface (S2S) radiation model, you can accelerate the calculation of the view factors with the following methods:

For the accelerated hemicube method, a combination of MPI/OpenMP/OpenCL models is used to speed up the view factor computations. Irrespective of the number of MPI processes launched, only one MPI process/machine is used for computing view factors. On each machine, one MPI process spawns several OpenMP threads that actually compute the view factors. The number of accelerated hemicube processes will be same as the number of Ansys Fluent processes. If OpenCL-capable GPUs are available, then a portion of the view factor computations are done on GPUs in offload mode using OpenCL, to further speed up the computation.

To use the accelerated hemicube method, set up the View Factors and Clustering dialog box and then enter the following text command to compute, write, and read a file with the surface cluster information and view factors: /define/models/radiation/s2s-parameters/compute-clusters-and-vf-accelerated. You will be prompted for the name of the file, the number of GPUs to be used, and the ratio of the work load on 1 GPU vs 1 CPU OpenMP thread (this is based on the time consumed by the GPU and the CPU). At the end of the view factor computations, a recommendation is printed for the GPU/CPU work load ratio to use in future computations.

Note that for the accelerated hemicube method, the OpenCL library should be accessible through the appropriate environment variable (LD_LIBRARY_PATH on lnamd64 or %path% on win64) in order to use the GPU for view factor computations. By default on lnamd64, /usr/lib64 is searched, but if the library is installed in another location, then that location should be specified in the LD_LIBRARY_PATH variable.

When using the accelerated ray tracing method with a GPU in offload mode, you must have one NVIDIA GPU (and only one) available on node-0. The NVIDIA Optix library is used for tracing the rays. Currently the Ampere GPUs are not supported with this method.

To use the accelerated ray tracing method, set up the View Factors and Clustering dialog box and then enter the following text command to compute, write, and read a file with the surface cluster information and view factors: /define/models/radiation/s2s-parameters/compute-clusters-and-vf-accelerated. You will be prompted for the name of the file.

The accelerated discrete ordinates (DO) radiation solver is computationally faster than the standard DO solver. Note that even though the accelerated DO solver may take more iterations to converge, the overall simulation time is shorter. If NVIDIA GPGPUs are set up in offload mode for the Fluent session, the accelerated DO solver will accelerate the DO computations by using the GPGPUs; in the absence of GPGPUs, this solver can still be used with the CPU cores to accelerate the DO computations.

After you have selected the DO model in the Radiation Model dialog box, you can enable the accelerated DO solver by using the following text command:

define → models → radiation → do-acceleration?

Note that the accelerated DO solver uses the first-order upwind scheme (and ignores whatever selection you have made for the Discrete Ordinates spatial discretization scheme in the Solution Methods task page), along with an explicit relaxation of 1.0.

If you plan to use GPGPUs with the accelerated DO solver, it is recommended that you run NVIDIA’s multi-process server (MPS) before launching Ansys Fluent using the following command: nvidia-cuda-mps-control -d. It is known to improve the robustness and performance of the GPGPU computations with the multiple Fluent processes. Then you must specify how many GPGPUs are to be used per machine when you launch Fluent: you can use the Solver GPGPUs per Machine (Offload Mode) setting in Fluent Launcher or use the -gpgpu= ngpgpus command line option. For details, refer to the following sections:

Note that using GPGPUs requires HPC licenses. Licensing details can be found in HPC Licensing in the Ansys, Inc. Licensing Guide.

Once the Fluent session is running, you can view and/or select the available GPGPUs on the system using the following TUI commands:

Currently the Ampere GPUs are not supported with the accelerated radiation solver on Windows platforms.