B.3. MPI Hydra Bootstrap Variable

On some Linux machines or clusters, you may find you need to set I_MPI_HYDRA_BOOTSTRAP for your parallel run to succeed. If you see an error message such as the one below, try setting this variable in either the Forte User Interface Linux Settings Panel or in your job submission script or environment.

	[mpiexec@hostname] check_exit_codes (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:117): unable to run bstrap_proxy on hostname(pid 8667, exit code 65280)
	[mpiexec@hostname] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:159): check exit codes error
	[mpiexec@hostname] HYD_dmx_poll_wait_for_proxy_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:212): poll for event error
	[mpiexec@hostname] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:1061): error waiting for event
	[mpiexec@hostname] HYD_print_bstrap_setup_error_message (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:1027): error setting up the bootstrap proxies
	[mpiexec@hostname] Possible reasons:
	[mpiexec@hostname] 1. Host is unavailable. Please check that all hosts are available.
	[mpiexec@hostname] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts. Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions.
	[mpiexec@hostname] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
	[mpiexec@hostname] 4. Ssh bootstrap cannot launch processes on remote host. Make sure that passwordless ssh connection is established across compute hosts.
	[mpiexec@hostname]    You may try using -bootstrap option to select alternative launcher.

On some clusters we have found that adding the environment variable still does not resolve the boostrap issue. If this is the case, remove the setting of the environment variable and instead add:

-bootstap ssh

to the mpirun run command. This can be done in the Forte UI on the Run Settings > Linux Settings > Parallel Execution Command as follows: