20.3. SLURM Queue

SLURM is a queuing system providing sbatch/squeue/scancel commands to submit, query, and delete the jobs.

FENSAP-ICE's graphical user interface will start a job with sbatch, creating and submitting a qsub.cfg job submission script file in the run folder. The command then returns a job ID, squeue is used to monitor the job and scancel is used to stop it, if requested.

Note:  squeue is used automatically as sqeue -o %.18i %8u %8t , and the supported output states for jobs of the current username are PD and CF (waiting in queue) and R (running). Any other state will not be processed by FENSAP-ICE's user interface, and the job is assumed to be ended or failed to start.

 

20.3.1. SLURM Job Settings

SLURM settings are specified in the submittal script via #SBATCH lines,these lines can be configured in FENSAP-ICE in the Configure… panel of the SLURM queue.

Refer to your queuing system documentation, in order to know which parameters to use here.

Figure 20.1: SLURM Settings Specified via #SBATCH Lines

SLURM Settings Specified via #SBATCH Lines

Queue name: Also known as the partition in SLURM. This is the job submission queue to use. This parameter must be configured. (Will translate to SBATCH -p)

  • Job name: The job name used for the registration in the queue. (SBATCH -J)

  • Walltime: Maximum execution time. (SBATCH -t)

  • SBATCH header: Additional commands added to the job submission script.

The I_MPI_PROCESS_MANAGER line will automatically tie the compute nodes provided by the queuing system with FENSAP-ICE MPI solvers. They will not require a -machinefile command line argument. Omit this line if you plan to use your own machinefile.

On certain SLURM systems you may need to specify the cores per compute node with the following flag:

#SBATCH --ntasks-per-node

 

20.3.2. Special Files

When submitting a PBS job, two files are created by FENSAP-ICE for your convenience:

  • slurm_job.txt

    Which contains the job ID, usually used with squeue.

  • slurm_nodes.txt

    Once the job is started, the qsub.cfg execution script will create this file and populate it with the list of all machines allocated by SLURM, one line per MPI count.

  • nodes.csv

    The slurm_nodes.txt is converted to a comma-separated-value list. This list can be useful for command line arguments of Fluent and CFX.

FENSAP-ICE will not read these files; they can be removed or modified.

 

20.3.3. CFX, FLUENT and SLURM

Fluent and CFX will not use automatically the MPI CPUs allocated by SLURM. Their command line argument should be configured appropriately in the configuration box for the flow solver, in the multi-shot.

The following arguments should be added:

  • Fluent: -cnf=slurm_nodes.txt

  • CFX: -par-dist `cat nodes.csv`

Note:  The backquote punctuation mark (`) is important as a single quote is not equivalent.