RSM provides built-in functionality that allows Workbench jobs to be submitted to a commercial cluster. The built-in functionality includes the ability to transfer files automatically to/from the cluster from a remote client and the ability to submit, cancel, and monitor Workbench jobs. The currently supported commercial clusters are Linux LSF, Linux PBS Pro, SLURM, Linux Altair Grid Engine (UGE), and Microsoft HPC (MSCC).
RSM also provides a custom cluster integration mechanism that allows third parties to use custom scripts to perform the tasks needed to integrate Workbench with the cluster. The custom integration scenarios can be grouped into the following categories in order of complexity:
Commercial clusters (listed above) for which you need some additional operation to be performed as part of the RSM job execution. This is a type of cluster-side integration.
“Unsupported” clusters, not included in the list above, that you want to use for executing a job via RSM. This is also a type of cluster-side integration.
You have specialized requirements that need to fully replace RSM functionality with 3rd-party scripts for handling all aspects of job submission, including file transfer. This is called client-side integration.
The terms cluster-side and client-side integration refer to the location (in the RSM architecture) where the custom script files are going to be located.
If the RSM client will be submitting jobs to a remote cluster, RSM files will be customized on the cluster submit host. This is referred to as cluster-side integration. For cluster-side integration, RSM must be installed on the cluster head node and file transfers should be handled by RSM, using either the internal RSM transfer mechanism or operating system file transfer to an existing network share. The methods of file transfer discussed in Setting Up Job Directories and File Transfers are available, except for Using SSH for File Transfers and Custom Client Integration.
If the RSM client is the cluster submit host, RSM files will be customized on the RSM client. This is referred to as client-side integration. In this scenario, the RSM functionality is completely replaced by the 3rd-party scripts. However, only a thin layer of the RSM architecture is involved, in order to provide the APIs for execution of the custom scripts, which are located on the client machine.
Note that for supported clusters it is also possible to include additional job submission arguments to the command executed by the cluster. The addition of custom submission arguments does not require the creation of custom scripts.
The following sections describe the general steps for customization with cluster-side and client-side integration. The detailed instructions for writing the custom code are similar for the two cases. They are addressed in Writing Custom Code for RSM Integration.
The following topics are addressed: