8.2.3. Configuring SSH/Custom File Transfers

If you are using SSH to transfer files to the HPC staging directory, you can customize commands to be used for client-to-HPC file management.

If you are using a custom mechanism for file transfers, you will need to create a custom HPC storage commands file. You will then need to add an entry to the jobConfiguration.xml file which references these files.

Follow the steps below to configure a customized file transfer mechanism.

  1. When creating a configuration, on the File Management tab, select External mechanism for file transfer (SCP, Custom), then select either SCP via SSH or Custom from the Transfer mechanism dropdown. Fill out the fields as described in File Management Properties for a Cluster.

    If you are using a Custom mechanism, make note of the keyword specified in the Custom transfer type field, as it will be used in upcoming steps.

  2. To define custom commands for transferring files to/from the HPC staging directory, edit or create a custom HPC storage commands file in the [RSMInstall]\Config\xml directory.

    • If you are using SSH-SCP, edit the hpc_storage_SSH.xml file. You may want to create a backup of the original file before editing it.

    • If you are using a custom mechanism, use hpc_storage_SSH.xml as a base to create a new HPC storage command file (for example, hpc_storage_AnsSecureTransfer.xml).

    Commands defined in this file include createStorage, deleteStorage, uploadToStorage, downloadFromStorage, and listStorageContents.

    Some commands use specific scripts for execution. For example, if you are using SSH, the uploadToStorage and downloadFromStorage commands use the scp.py script. If you are using a custom mechanism, you will need to create custom command scripts and reference them in the command definitions in the HPC storage commands file. For example, if your custom keyword is AnsSecureTransfer, you might create a script called AnsSecureTransfer.py, and reference that script in the uploadToStorage and downloadFromStorage commands.

    To customize command scripts, go to the [RSMInstall]\Config\scripts directory.

    RSM will set the following variables when file transfer commands are executed:

    • RSM_HPC_FILEDIRECTION

    • RSM_HPC_FILECONTEXT

    • RSM_HPC_FILELIST

    • RSM_HPC_FILEUPLOADTRACKING

    • RSM_HPC_FILE_LOCAL_DIR

    • RSM_HPC_FILE_REMOTE_DIR

    • RSM_HPC_FILEEXCLUDE (if exclusions exist)

    For more information about these variables, see Environment Variables Set by RSM.

  3. If you are using a custom mechanism, edit the jobConfiguration.xml file in the [RSMInstall]\Config\xml directory.

    Create an entry for your custom transfer type which references the custom HPC storage commands file (and optionally the custom protocol definition file) that you created in the previous steps. Be sure to append your keyword (that you specified in step 1) to the keyword name. For example:

    <keyword name="STORAGE_AnsSecureTransfer">
    <hpcCommands name="hpc_storage_AnsSecureTransfer.xml"/>
  </keyword>

    Note that there is already an entry for the SSH-SCP protocol in the jobConfiguration.xml file named STORAGE_SSH-SCP.