Monitoring Slurm Cluster Jobs
To check whether a Slurm cluster is ready or get information on batch jobs that have been submitted to the cluster, you can issue Slurm commands at a command prompt on the Slurm Controller machine. For an autoscaling cluster, the Slurm Controller runs on the cluster head node. See Connecting to an Autoscaling Cluster Head Node.
The two commands used to get cluster and job information, sinfo and squeue, are described below. For a full list of Slurm commands, go to https://slurm.schedmd.com/quickstart.html.
Slurm Cluster State (sinfo)
To check whether a Slurm cluster is fully defined and ready, use the sinfo command.
If the cluster is ready, the number of nodes reported by sinfo will be equal to what you requested. If it is not ready, wait a few minutes and then try sinfo again. If there is a delay, typically it is only 1 to 2 minutes.
For more information about the sinfo command, go to https://slurm.schedmd.com/sinfo.html.
Slurm Job State (squeue)
A batch job goes through several states in the course of its execution. To view the current state of a batch job managed by Slurm, use the squeue command.
In the command output, the job state is indicated in the 'ST' column.
Typical job states are described below.
| Job state in output | Meaning | Description |
|---|---|---|
| PD | Pending | The job is queued. It is waiting for resource allocation. |
| R | Running | The job is allocated to a node and is running. |
| CG | Completing | The job is finished but some processes on some nodes may still be active. |
| CD | Completed | The job has completed successfully. |
| F | Failed | The job has failed. |
| TO | Timeout | The job has reached its runtime limit and has been terminated by Slurm. |
For more information about the squeue command and a complete list of possible job states, go to https://slurm.schedmd.com/squeue.html.