When a job is launched using the PBS queuing system, PBS will provide a list of machines for MPI execution. This list must be provided to Additional mpirun parameters with the following argument in the advanced settings:

This option is accessed with the Configure button in the execution window, as shown in the preceding section. The setting should be saved before proceeding with the execution.

This setting can also be used as a system-wide default in $NTI_PATH/../config/mpi.txt.

In the Configure queue dialog window, some general settings can be specified:

Queue name

This option might be required if there are multiple queues available on the computer. If specified, the queue name will be translated into the PBS command:

#PBS –q NAME

Nodes ppn

Some queuing systems require the number of processors per node (ppn). For example, on a cluster composed of quad-core processors, the ppn value would be set to 4. Therefore, to run an 8-CPU job making efficient use of the resources one must specify two nodes with ppn=4.

#PBS -l nodes=2:ppn=8

If not specified, the ppn default value for PBS is 1. For example:

If PPN is required by the queue, or if the job must run on a specific number of processors on each node, use the Nodes ppn option.

Example 20.2: 64-processor machine with 8 processors per node

A 32-CPU job on this machine would translate to:

#PBS -l nodes=4:ppn=8

Walltime

Walltime is the maximum time for job execution. It might be mandatory on the PBS system. This setting will be supplied to PBS in the format:

#PBS -l walltime=24:00:00

All the PBS settings described here will be written to the qsub.cfg file in the run directory. Therefore, when executing a FENSAP-ICE module, the file:

$HOME/../Project/run_FENSAP/qsub.cfg will be created. This file is only a PBS wrapper for the .solvercmd shell script. It may contain, for example:

#!/bin/sh -f

#PBS -N fensap

#PBS -l ncpus=2

#PBS -V

cd /home/user/Project/run_FENSAP/

rm -f fensap.exitStatus fensap.stdout

cp $PBS_NODEFILE pbs_nodes.txt

/home/user/Project/run_FENSAP/.solvercmd 2>&1 > fensap.stdout

echo $? > fensap.exitStatus

The contents of the file can be customized by using the Advanced settings section of the PBS configuration:

PBS header: Commands entered here will be added at the top of the qsub.cfg file.

Job init: Commands entered here will be executed before .solvercmd execution.

Job end: Commands entered here will be executed after .solvercmd execution.

When submitting a PBS job, two files are created by FENSAP-ICE for your convenience:

pbs_job.txt

Which contains the job ID, usually used with qstat.

pbs_nodes.txt

Once the job has started, the qsub.cfg execution script will create this file, containing the content of $PBS_NODEFILE.

FENSAP-ICE will not read these files; they can be removed or modified.