When pdsh receives SIGINT (ctrl-C), it lists the status of current threads. A second SIGINT within one second terminates the program. Pending threads may be canceled by issuing ctrl-Z within one second of ctrl-C. Pending threads are those that have not yet been initiated, or are still in the process of connecting to the remote host.

If a remote command is not specified on the command line, pdsh runs interactively, prompting for commands and executing them when terminated with a carriage return. In interactive mode, target nodes that time out on the first command are not contacted for subsequent commands, and commands prefixed with an exclamation point will be executed on the local system.

The core functionality of pdsh may be supplemented by dynamically loadable modules. The modules may provide a new connection protocol (replacing the standard rcmd(3) protocol used by rsh(1)), filtering options (e.g. removing hosts that are "down" from the target list), and/or host selection options (e.g., -a selects all hosts from a configuration file.). By default, pdsh must have at least one "rcmd" module loaded. See the RCMD MODULES section for more information.

A list of rcmd modules currently distributed with pdsh follows.

rsh

Uses an internal, thread-safe implementation of BSD rcmd(3) to run commands using the standard rsh(1) protocol.

ssh

Uses a variant of popen(3) to run multiple copies of the ssh(1) command.

mrsh

This module uses the mrsh(1) protocol to execute jobs on remote hosts. The mrsh protocol uses a credential based authentication, forgoing the need to allocate reserved ports. In other aspects, it acts just like rsh. Remote nodes must be running mrshd(8) in order for the mrsh module to work.

qsh

Allows pdsh to execute MPI jobs over QsNet. Qshell propagates the current working directory, pdsh environment, and Elan capabilities to the remote process. The following environment variable are also appended to the environment: RMS_RANK, RMS_NODEID, RMS_PROCID, RMS_NNODES, and RMS_NPROCS. Since pdsh needs to run setuid root for qshell support, qshell does not directly support propagation of LD_LIBRARY_PATH and LD_PREOPEN. Instead the QSHELL_REMOTE_LD_LIBRARY_PATH and QSHELL_REMOTE_LD_PREOPEN environment variables will may be used and will be remapped to LD_LIBRARY_PATH and LD_PREOPEN by the qshell daemon if set.

mqsh

Similar to qshell, but uses the mrsh protocol instead of the rsh protocol.

krb4

The krb4 module allows users to execute remote commands after authenticating with kerberos. Of course, the remote rshd daemons must be kerberized.

xcpu

The xcpu module uses the xcpu service to execute remote commands.

-w [rcmd_type:][user@]host,host,...

Target the specified list of hosts. Do not use with any other node selection options (e.g. -a, -g if they are available). No spaces are allowed in the comma-separated list. A list consisting of a single `-' character causes the target hosts to be read from stdin, one per line. The host list may contain hostlist expressions of the form ``host[1-5,7]''. For more information about the hostlist format, see the HOSTLIST EXPRESSIONS section below. A list of hosts may also be preceded by "user@" to specify a remote username other than the default, or "rcmd_type:" to specify an alternate rcmd connection type for these hosts. When used together, the rcmd type must be specified first, e.g. "ssh:user1@host0" would use ssh to connect to host0 as user "user1."

-x host,host,...

Exclude the specified hosts. May be specified in conjunction with other target node list options such as -a and -g (when available). Hostlists may also be specified to the -x option (see the HOSTLIST EXPRESSIONS section below).

-S

Return the largest of the remote command return values.

-h

Output usage menu and quit. A list of available rcmd modules will also be printed at the end of the usage message.

-s

Only on AIX, separate remote command stderr and stdout into two sockets.

-q

List option values and the target nodelist and exit without action.

-b

Disable ctrl-C status feature so that a single ctrl-C kills parallel job. (Batch Mode)

-l user

This option may be used to run remote commands as another user, subject to authorization. For BSD rcmd, this means the invoking user and system must be listed in the user's .rhosts file (even for root).

-t seconds

Set the connect timeout. Default is 10 seconds.

-u seconds

Set a limit on the amount of time a remote command is allowed to execute. Default is no limit. See note in LIMITATIONS if using -u with ssh.

-f number

Set the maximum number of simultaneous remote commands to number. The default is 32.

-R name

Set rcmd module to name. This option may also be set via the PDSH_RCMD_TYPE environment variable. A list of available rcmd modules may be obtained via the -h, -V, or -L options. The default will be listed with -h or -V.

-L

List info on all loaded pdsh modules and quit.

-d

Include more complete thread status when SIGINT is received, and display connect and command time statistics on stderr when done.

-V

Output pdsh version information, along with list of currently loaded modules, and exit.

-n tasks_per_node

Set the number of tasks spawned per node. Default is 1.

-m block | cyclic

Set block versus cyclic allocation of processes to nodes. Default is block.

-r railmask

Set the rail bitmask for a job on a multirail system. The default railmask is 1, which corresponds to rail 0 only. Each bit set in the argument to -r corresponds to a rail on the system, so a value of 2 would correspond to rail 1 only, and 3 would indicate to use both rail 1 and rail 0.

-a

Target all nodes from machines file.

In addition to the genders options presented below, the genders attribute pdsh_rcmd_type may also be used in the genders database to specify an alternate rcmd connect type than the pdsh default for hosts with this attribute. For example, the following line in the genders file

  host0 pdsh_rcmd_type=ssh

-A

Target all nodes in genders database. The -A option will target every host listed in genders -- if you want to omit some hosts by default, see the -a option below.

-a

Target all nodes in genders database except those with the "pdsh_all_skip" attribute. This is shorthand for running "pdsh -A -X pdsh_all_skip ..."

-g attr[=val][,attr[=val],...]

Target nodes that match any of the specified genders attributes (with optional values). Conflicts with -a and -w options. This option targets the alternate hostnames in the genders database by default. The -i option provided by the genders module may be used to translate these to the canonical genders hostnames. If the installed version of genders supports it, attributes supplied to -g may also take the form of genders queries. Genders queries will query the genders database for the union, intersection, difference, or complement of genders attributes and values. The set operation union is represented by two pipe symbols ('||'), intersection by two ampersand symbols ('&&'), difference by two minus symbols ('--'), and complement by a tilde ('~'). Parentheses may be used to change the order of operations. See the nodeattr(1) manpage for examples of genders queries.

-X attr[=val][,attr[=val],...]

Exclude nodes that match any of the specified genders attributes (optionally with values). This option may be used in combination with any other of the node selection options (e.g. -w, -g, -a, -X may also take the form of genders queries. Please see documentation for the genders -g option for more information about genders queries.

-i

Request translation between canonical and alternate hostnames.

-F filename

Read genders information from filename instead of the system default genders file.

-v

Eliminate target nodes that are considered "down" by libnodeupdown.

-j jobid[,jobid,...]

Target list of nodes allocated to the SLURM job jobid. This option may be used multiple times to target multiple SLURM jobs.

-a

Target all nodes in the SDR. The list is generated from the "reliable hostname" in the SDR by default.

-i

Translate hostnames between reliable and initial in the SDR, when applicable. If the a target hostname matches either the initial or reliable hostname in the SDR, the alternate name will be substitued. Thus a list composed of initial hostnames will instead be replaced with a list of reliable hostnames. For example, when used with -a above, all initial hostnames in the SDR are targeted.

-v

Do not target nodes that are marked as not responding in the SDR on the targeted interface. (If a hostname does not appear in the SDR, then that name will remain in the target hostlist.)

-G

In combination with -a, include all partitions.

-g groupname,...

Target nodes in dsh group file "groupname" found in either ~/.dsh/group/groupname or /etc/dsh/group/groupname.

-X groupname,...

Exclude nodes in dsh group file "groupname."

-g groupname,...

Target nodes in netgroup "groupname."

-X groupname,...

Exclude nodes in netgroup "groupname."

PDSH_RCMD_TYPE

Equivalent to the -R option, the value of this environment variable will be used to set the default rcmd module for pdsh to use (e.g. ssh, rsh).

PDSH_SSH_ARGS

Override the standard arguments that pdsh passes to the ssh(1) command ("-2 -a -x").

PDSH_SSH_ARGS_APPEND

Append additional options to the ssh(1) command invoked by pdsh. For example, PDSH_SSH_ARGS_APPEND="-q" would run ssh in quiet mode, or "-v" would increase the verbosity of ssh.

WCOLL

If no other node selection option is used, the WCOLL environment variable may be set to a filename from which a list of target hosts will be read. The file should contain a list of hosts, one per line (though each line may contain a hostlist expression. See HOSTLIST EXPRESSIONS section below).

DSHPATH

If set, the path in DSHPATH will be used as the PATH for the remote processes.

FANOUT

Set the pdsh fanout (See description of -f above).

The hostlist syntax is meant only as a convenience on clusters with a "prefixNNN" naming convention and specification of ranges should not be considered necessary -- this foo1,foo9 could be specified as such, or by the hostlist foo[1,9].

Some examples of usage follow:

Run command on foo01,foo02,...,foo05
    pdsh -w foo[01-05] command

Run command on foo7,foo9,foo10
        pdsh -w foo[7,9-10] command

Run command on foo0,foo4,foo5
        pdsh -w foo[0-5] -x foo[1-3] command

As a reminder to the reader, some shells will interpret brackets ('[' and ']') for pattern matching. Depending on your shell, it may be necessary to enclose ranged lists within quotes. For example, in tcsh, the first example above should be executed as:

pdsh -w "foo[01-05]" command

When using ssh for remote execution, expect the stderr of ssh to be folded in with that of the remote command. When invoked by pdsh, it is not possible for ssh to prompt for passwords if RSA/DSA keys are configured properly, etc.. Additionally, the connect timeout is not adjustable when ssh is used. Finally, there is no reliable way for pdsh to ensure that remote commands are actually terminated when using a command timeout. Thus if -u is used with ssh commands may be left running on remote hosts even after timeout has killed local ssh processes.

Output from multiple processes per node may be interspersed when using qshell or mqshell rcmd modules.

Hostlist parsing assumes numerical part of hostname is at the end only, e.g., specifying foo[0-5]bar will not work.

The number of nodes that pdsh can simultaneously execute remote jobs on is limited by the maximum number of threads that can be created concurrently, as well as the availability of reserved ports in the rsh and qshell rcmd modules. On systems that implement Posix threads, the limit is typically defined by the constant PTHREADS_THREADS_MAX.