Upon startup and during a DMP solution, the program appends n to the current jobname (where n stands for the process rank). The master process rank is 0 and the worker processes are numbered from 1 to N - 1. In the rare case that the user-supplied jobname ends in a numeric value (0...9) or an underscore, an underscore is automatically appended to the jobname prior to appending the process rank. This is done to avoid any potential conflicts with the new file names.

Therefore, upon startup and during a parallel solution, each process will create and use files named Jobnamen.ext. These files contain the local data that is specific to each DMP process. Some common examples include the .log and .err files as well as most files created during solution such as .esav, .full, .rst, and .mode. See Program-Generated Files in the Basic Analysis Guide for more information on files that Mechanical APDL typically creates.

Actions that are performed only by the master process (/PREP7, /POST1, etc.) will work on the global Jobname.ext files by default. These files (such as Jobname.db and Jobname.rst) contain the global data for the entire model. For example, only the master process will save and resume the Jobname.db file.

After a parallel solution successfully completes, the program automatically merges some of the (local) Jobnamen.ext files into a single (global) file named Jobname.ext. These include files such as Jobname.rst, Jobname.esav, Jobname.emat, and so on (see the DMPOPTION command for a complete list of these files). This action is performed when the FINISH command is executed upon leaving the solution processor. These files contain the same information about the final computed solution as files generated for the same model computed with SMP processing. Therefore, all downstream operations (such as postprocessing) can be performed using SMP processing (or in the same manner as SMP processing) by using these global files.

If any of these global Jobname.EXT files are not needed for downstream operations, you can reduce the overall solution time by suppressing the file combination for individual file types (see the DMPOPTION command for more information). If it is later determined that a global Jobname.EXT file is needed for a subsequent operation or analysis, the local files can be combined by using the COMBINE command.

DMPOPTION also has an option to combine the results file at certain time points during the distributed solution. This enables you to postprocess the model while the solution is in progress, but leads to slower performance due to increased data communication and I/O.

The program will not delete most files written by the worker processes when the analysis is completed. If you choose, you can delete these files when your analysis is complete (including any restarts that you may wish to perform). If you do not wish to have the files necessary for a restart saved, you can issue RESCONTROL,NORESTART.

File copy, delete, and rename operations can be performed across all processes by using the DistKey option on the /COPY, /DELETE, and /RENAME commands. This provides a convenient way to manage local files created by a distributed parallel solution. For example, /DELETE,Fname,ext,,ON automatically appends the process rank number to the specified file name and deletes Fnamen.ext from all processes. If the local files (Jobname.*) are not needed in downstream analyses, you can issue /FCLEAN to delete them all to save space. See the /COPY, /DELETE, /FCLEAN, and /RENAME command descriptions for more information. In addition, the /ASSIGN command can be used to control the name and location of specific local and global files created during a distributed-memory parallel analysis.

You can launch a DMP analysis in either interactive or batch mode for the master process. However, the worker processes are always in batch mode. The worker processes cannot read the start.ans or stop.ans files. The master process sends all /CONFIG,LABEL commands to the worker processes as needed.

On Windows systems, there is no Mechanical APDL output console window when running a DMP analysis in the GUI (interactive mode). All standard output from the master process will be written to a file named file0.out. (Note that the Jobname is not used.)

When a DMP analysis is executed, the output for the master process is written in the same fashion as an SMP run. By default, the output is written to the screen; or, if you specified an output file via the launcher or the -o command line option, the output for the master process is written to that file.

The worker processes automatically write this output to Jobnamen.out. Typically, these worker process output files have little value because all of the relevant job information is written to the screen or the master process output file. The exception is when the domain decomposition method is automatically chosen to be FREQ (frequency-based for a harmonic analysis) or CYCHI (harmonic index-based for a cyclic symmetry analysis); see the DDOPTION command for more details. In these cases, the solution information for the harmonic frequencies or cyclic harmonic indices solved by the worker processes are only written to the output files for those processes (Jobnamen.out).

The same principle also applies to the error file Jobnamen.err. When a warning or error occurs on one of the worker processes during a DMP solution, the process writes that warning or error message to its error file and then communicates the warning or error message to the master process. Typically, this allows the master process to write the warning or error message to its error file and output file and, in the case of an error message, allows all of the distributed processes to exit the program simultaneously.

In some cases, an error message may fail to be fully communicated to the master process. If this happens, you can view each Jobnamen.err and/or Jobnamen.out file in an attempt to learn why the job failed. The error files and output files written by all the processes will be incomplete but may still provide some useful information as to why the job failed.

In some rare cases, the job may hang. When this happens, you can use the cleanup script (or .bat file on Windows) to kill the processes. The cleanup script is automatically written into the initial working directory of the master process and is named cleanup-ansys-[machineName]-[processID].sh (or .bat).

In pre- and postprocessing, APDL works the same in a DMP analysis as in an SMP analysis. However, in the solution processor (/SOLU), a DMP analysis does not support certain *GET items. In general, DMP processing supports global solution *GET results such as total displacements and reaction forces. It does not support element level results specified by ESEL, ESOL, and ETABLE labels. Unsupported items will return a *GET value of zero.

Multiple commands entered in an input file can be condensed into a single line if the commands are separated by the $ character (see Condensed Data Input in the Command Reference). DMP processing cannot properly handle condensed data input. Each command must be placed on its own line in the input file for a DMP run.

Upon starting a solution in a DMP analysis, the program automatically decomposes the problem into N CPU domains so that each process (or each CPU core) works on only a portion of the simulation. This domain decomposition is done only by the master process. Typically, the optimal domain decomposition method is automatically chosen based on a variety of factors (analysis type, number of CPU cores, available RAM, and so on). However, you can control which domain decomposition method is used by setting the Decomp argument on the DDOPTION command.

For most simulations, the program automatically chooses the mesh-based domain decomposition method (Decomp = MESH), which means each CPU domain is a group or subset of elements within the whole model. For certain harmonic analyses, the domain decomposition may be based on the frequency domain (Decomp = FREQ), in which case each CPU domain computes the harmonic solution for the entire model at a different frequency point. For certain cyclic symmetry analyses, the domain decomposition may be based on the harmonic indices (Decomp = CYCHI), in which case each CPU domain computes the cyclic solution for a different harmonic index.

The NPROCPERSOL argument on the DDOPTION command gives you the flexibility to combine the FREQ or CYCHI decomposition methods with the mesh-based domain decomposition method. Consider a harmonic analysis with 50 frequency points requested (NSUBST,50) run on a workstation executing a DMP solution with 16 cores (-dis-np 16). Using mesh decomposition (DDOPTION,MESH) essentially solves one frequency at a time with 16 groups of elements (1x16). Using DDOPTION,FREQ,1 solves 16 frequencies at a time with 1 group of elements; that is, the entire FEA model (16x1). Using the NPROCPERSOL field allows you to consider alternative combinations in between these 2 scenarios. You could try DDOPTION,FREQ,2 to solve 8 frequencies at a time with 2 groups of elements per solution (8x2), or DDOPTION,FREQ,4 to solve 4 frequencies with 4 groups of elements (4x4), and so on. Note that the total core count specified at startup cannot be altered, and the program works to maintain the NPROCPERSOL value as input, which means the number of frequency or cyclic harmonic index solutions solved at a time may need to be adjusted to fit within the other defined parameters.

In the case of a linear perturbation analysis, you should also set the NUMSOLFORLP argument to ensure the best solution performance. For decomposition based on the frequency domain (Decomp = FREQ or automatically chosen when Decomp = AUTO), set NUMSOLFORLP to the number of frequency solutions in the subsequent harmonic analysis. For decomposition based on harmonic index (Decomp = CYCHI or automatically chosen when Decomp = AUTO), set NUMSOLFORLP to the number of harmonic index solutions in the subsequent cyclic modal analysis.

There are pros and cons for each domain decomposition approach. For example, with the default MESH method, the total amount of memory required to solve the simulation in N processes is typically not much larger than the amount of memory required to solve the simulation on one process. However, when using the FREQ or CYCHI domain decomposition method, the amount of memory required to solve the simulation on N processes is typically N times greater than the amount of memory to solve the simulation on one process. As another example, with the MESH domain decomposition method, the application requires a significant amount of data communication via MPI, thus requiring a fast interconnect for optimal performance. With the FREQ and CYCHI methods, very little data communication is necessary between the processes and, therefore, good performance can still be achieved using slower interconnect hardware.

All FEA data (elements, nodes, materials, sections, real constants, boundary conditions, etc.) required to compute the solution for each CPU domain is communicated to the worker processes by the master process. Throughout the solution, each process works only on its piece of the entire model. When the solution phase ends (for example, FINISH is issued in the solution processor), the master process in a DMP analysis works on the entire model again (that is, it behaves like SMP processing).

In DMP processing, the OUTPR command prints NSOL and RSOL in the same manner as in SMP processing. However, for other items such as ESOL, a DMP analysis prints only the element solution for the group of elements belonging to the CPU domain of the master process. Therefore, OUTPR, ESOL has incomplete information and is not recommended. Also, the order of elements is different from that of SMP processing due to domain decomposition. A direct one-to-one element comparison with SMP processing will be different if using OUTPR.

When using the frequency domain decomposition in a harmonic analysis, the NSOL, RSOL, and ESOL data is printed into the output file written by the master process that computes the frequency solutions. The OUTPR print for the other frequency solutions is written to the output files created by the worker processes (Jobnamen.out).

Both SMP processing and DMP processing can handle a large number of coupling and constraint equations (CE/CP) and contact elements. However, specifying too many of these items can force a DMP solution to communicate more data among each process, resulting in longer elapsed time to complete a distributed parallel job. You should reduce the number of CE/CP if possible and make potential contact pairs in a smaller region to achieve non-deteriorated performance. In addition, for assembly contact pairs or small sliding contact pairs, you can use the command CNCHECK,TRIM to remove contact and target elements that are initially in far-field (open and not near contact). This trimming option will help to achieve better performance in DMP runs.

SMP processing can postprocess the last set of results using the Jobname.db file (if the solution results were saved), as well as using the Jobname.rst file. A DMP analysis, however, can only postprocess using the Jobname.rst file and cannot use the Jobname.db file as solution results are not entirely written to the database. You will need to issue a SET command before postprocessing.

By default, a DMP analysis will automatically combine the local results files (for example, Jobnamen.rst) into a single global results file (Jobname.rst). This step can be expensive depending on the number of load steps and the amount of results stored for each solution. It requires each local results file to be read by each worker process, communicated to the master process, and then combined together and written by the master process. As a means to reduce the amount of communication and I/O performed by this operation, the DMPOPTION command can be used to skip the step of combining the local results file into a single global results file. Then the RESCOMBINE command macro can be used in /POST1 to individually read each local results file until the entire set of results is placed into the database for postprocessing. If needed, a subsequent RESWRITE command can then be issued to write a global results file for the distributed solution.

Note that if the step of combining the results file is skipped, it may affect downstream analyses that rely on a single global results file for the entire model. If it is later determined that a global results file (for example, Jobname.rst) is needed for a subsequent operation, you can use the COMBINE command to combine the local results files into a single, global results file.