23.3. Cray Message Passing Interface

The use of the Cray message passing interface (MPI) for parallel simulations on Linux is supported as a beta feature. To use this MPI, include the following command line option when launching Fluent: -mpi=cray.

Note the following:

  • Node processes on the same machine communicate by shared memory. Ansys Fluent lets the MPI autoselect the best interconnect available on the system. If you want to specify an interconnect to override that selection, you can use the -p<interconnect> command line option. The crayx interconnect is available for Cray systems running Cray Message Passing Toolkit (MPT) 7.0. Then the MPI is selected automatically, based on the specified interconnect.

  • Note that on systems using large pages for memory allocation (such as Cray systems), the virtual memory usage reported by Fluent may be much higher than actual memory used. In this case resident memory (also reported by Fluent) is a more reliable guide.

  • Interpreted user-defined functions (UDFs) cannot be used on the Cray platform when running in parallel. The compiled UDF approach must be used instead.

  • Dynamic process spawning is not supported on Cray systems: when switching from meshing mode to solution mode, Fluent will not automatically spawn additional parallel node processes to achieve the requested number of total solution processes. While you can keep the same number of cores for meshing mode as for solver mode, it is recommended that you continue the solver iterations using another separate solver session.

  • Note that when using MPT version 5.0 and higher (up to, but not including 7.0), you must set the following environment variable: export FLUENT_USE_CRAY_MPT5=1.

  • When creating simulation reports on the Cray platform, Fluent may report an error due to one or more missing Nexus libraries.

  • When starting remote visualization, you must start the server and the remote client. Note that for clusters with isolated compute nodes (such as a Cray system) you must use the export REMOTING_SERVER_ADDRESS environment variable to provide an outward facing IP address for the client to connect to. This environment variable must be defined on both the server and the client.

  • When using the integrated PBS Professional capability in Fluent, note that while RMS-specific checkpointing is not available, PBS Professional does support checkpointing via mechanisms that are specific to the operating system on Cray systems. The integrated PBS Professional capability is based on saving the process state, and is not based on the standard application restart files (for example, Fluent case and data files) on which the LSF and SGE checkpointing is based. Thus, if you need to checkpoint, you should checkpoint your jobs by periodically saving the Fluent data file via the journal file.