tsshbatch - Run Commands On Batches Of Machines
tsshbatch is a powerful tool for automating activities
on many servers at a time. This also gives you to power
to make many mistakes at a time! This is especially
true if you have
sudo privilege promotion
capabilities on the systems in your care. So be careful
We therefore STRONGLY recommend you do the following
things to mitigate this risk:
- Read This Fine Manual from beginning to end.
- Practice using
tsshbatch on test machines or
VMs that can easily be recovered or reimaged if you
- Make heavy use of test mode (which is the default)
to see what the program would do if it actually
ran in execution mode.
tsshbatch.py [-EKNSTaehkvxy -G 'file dest' -P 'file dest' -f cmdfile -l logfile -n name -p pw ] -H 'host ..' | hostlistfile [command arg ... ]
tsshbatch is a tool to enable you to issue a command to many
hosts without having to log into each one separately. When writing
scripts, this overcomes the
ssh limitation of not being able to
specify the password on the command line.
You can also use
from- and to many hosts at once.
tsshbatch also understands basic
sudo syntax and can be used
to access a host,
sudo a command, and then exit.
tsshbatch thus allows you to write complex, hands-off scripts that
issue commands to many hosts without the tedium of manual login and
sudo promotion. System administrators, especially, will find this
helpful when working in large server farms.
tsshbatch supports a variety of options which can be specified
on either the command line or in the
tsshbatch writes it's own errors
stderr. It also writes the
output from each host it contacts to the local
stderr (unless the
-e option has
-E option redirects any such
output intended for
instead. This avoids the need to do things like
2>&1 | ...` on the command line when you want to
pipe all ``tsshbatch output to another program.
||Force prompting for passwords. This is used
to override a prior
GET file on host and write local dest directory.
spec is a quoted pair of strings. The first
specifies the path of the source file (on the
remote machine) to copy. The second, specifies
the destination directory (on the local
tsshbatch.py -G "/foo/bar/baz /tmp" hostlist
/foo/bar/baz from every machine in
hostlistfile to the local
Since all the files have the same name, they would
overwrite each other if copied into the same
tsshbatch prepends the string
hostname- to the name of each file it saves
List of hosts on which to run the command. This should
be enclosed in quotes so that the list of hosts
is handed to the -H option as a single argument:
-H 'host1 host2 host3'
||Force interactive username dialog. This cancels
any previous request for key exchange authentication.
PUT file from local machine to remote machine destination
spec is a quoted pair of strings.
The first specifies the path of the source file (on
the local machine) to copy. The second, specifies
the destination directory (on the remote machine):
tsshbatch.py -P "/foo/bar/baz /tmp" hostlist
/foo/bar/baz on the local
/tmp/ on every host in
||Force prompting for sudo password
||Timeout for ssh connection attempts (Default: 15 seconds)
||Don't abort program after failed file transfers.
Continue to next transfer attempt. (Default: Abort)
||Don't report remote host
||Read commands from a file. This file can be commented
freely with the
# character. Leading- and
trailing whitespace on a line are ignored.
||Print help information
||Use ssh keys instead of name/password credentials
||Log diagnostic output to
logfile (Default: /dev/null)
||Login name to use
||Password to use when logging in and/or doing sudo
||Test mode: Only show what would be done but don't
actually do it. This also prints diagnostic
information about any variable definitions, the list
of hosts, any
PUT requests, and final
command strings after all variable substitutions have
been applied. This is the default program behavior.
||Print detailed program version information and exit
||Override any previous
-t specifications and
actually execute the commands. This is useful
if you want to put
-t in the
environment variable so that the default is
always run the program in test mode. Then, when
you're ready to actually run commands, you can
override it with
-x on the command line.
||Turn on 'noisy' reporting for additional detail on
every line, instead of just at the top of the
stderr reporting. This is
helpful when you are filtering the output through
grep that only returns matching
lines and thus no context information.
-H option is not selected, the item immediately following
the options is understood to be the name of the
This is a file that contains the name of each host - one per line - on
which to run the commands. This file can be commented freely with the
# character. Leading- and trailing whitespace on a line are
The last entry on the command line is optional and defines a command
tsshbatch will attempt to execute it on every host you've
specified either via
-H or a
tsshbatch.py -Hmyhost ls -al /etc
This will do a
ls -al /etc on
Be careful when using metacharacters like
&&, <<, >>, <, > and so
on in your commands. You have to escape and quote them properly or
your local shell will interfere with them being properly conveyed to
the remote machine.
If you've specified a
cmdfile containing the commands you want run via
-f option, these commands will run before the command
you've defined on the command line. It is always the last command
run on each host.
You can put as many
-f arguments as you wish on the command
line and the contents of these files will be run in the order
they appeared from left-to-right on the command line.
tsshbatch does all the
GETs, then all the
attempting to do any command processing. If no
commands have been specified,
tsshbatch will exit silently, since
"nothing to do" really isn't an error.
tsshbatch respects the
$TSSHBATCH environment variable. You
may set this variable with any options above you commonly use to avoid
having to key them in each time you run the program. For example:
export TSSHBATCH="-n jluser -p l00n3y"
This would cause all subsequent invocations of
attempt to use the login name/password credentials of
tsshbatch also supports searching for files over specified
paths with the
variables. Their use is described later in this document.
FEATURES AND USE CASES
Different Ways To Specify Targeted Hostnames
There are two ways to specify the list of hosts on which you want
to run the specified command:
On the command line via the
tsshbatch.py -H 'hostA hostB' uname -a
This would run the command
uname -a on the
Notice that the list of hosts must be separated by spaces but
passed as a single argument. Hence we enclose them in single
Via a host list file:
tsshbatch.py myhosts df -Ph
tsshbatch expects the file
myhosts to contain a
list of hosts, one per line, on which to run the command
-Ph. As an example, if you want to target the hosts
myhosts would look
This method is handy when there are standard "sets" of hosts on
which you regularly work. For instance, you may wish to keep a
host file list for each of your production hosts, each of your
test hosts, each of your AIX hosts, and so on.
You may use the
# comment character freely throughout a
host list file to add comments or temporarily comment out a
particular host line.
You can even use the comment character to temporarily comment
out one or most hosts in the list given to the
line argument. For example:
tsshbatch.py -H "foo #bar baz" ls
This would run the
ls command on hosts
bar. This is handy if you want to use your
shell's command line recall to save typing but only want to
repeat the command for some of the hosts your originally
Authentication Using Name And Password
The simplest way to use
tsshbatch is to just name the hosts
can command you want to run:
tsshbatch.py linux-prod-hosts uptime
tsshbatch uses your login name found in the
$USER environment variable when logging into other systems.
In this example, you'll be prompted only for your password which
tsshbatch will then use to log into each of the machines named
linux-prod-hosts. (Notice that his assumes your name and
password are the same on each host!)
Typing in your login credentials all the time can get tedious after
tsshbatch provides a means of providing them on the
tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime
This allows you to use
tsshbatch inside scripts for hands-free
If your login name is the same on all hosts, you can simplify
this further by defining it in the environment variable:
export TSSHBATCH="-n joe.luser"
Any subsequent invocation of
tsshbatch will only require a
password to run.
HOWEVER, there is a huge downside to this - your plain text
password is exposed in your scripts, on the command line, and
possibly your command history. This is a pretty big security hole,
especially if you're an administrator with extensive privileges.
(This is why the
ssh program does not support such an option.)
For this reason, it is strongly recommended that you use the
option sparingly, or not at all. A better way is to push ssh keys
to every machine and use key exchange authentication as described
However, there are times when you do have use an explicit password,
such as when doing
sudo invocations. It would be really nice
-p and avoid having to constantly type in the password.
There are two strategies for doing this more securely than just
entering it in plain text on the command line:
Temporarily store it in the environment variable:
export TSSHBATCH="-n joe.luser -p my_weak_pw"
Do this interactively after you log in, not from
a script (otherwise you'd just be storing the plain text
password in a different script). The environment variable
will persist as long as you're logged in and disappear
when you log out.
If you use this just make sure to observe three security
- Clear your screen immediately after doing this so no one
walking by can see the password you just entered.
- Configure your shell history system to ignore
commands beginning with
export TSSHBATCH. That
way your plain text password will never appear in
the shell command history.
- Make sure you don't leave a logged in session unlocked so
that other users could walk up and see your password by
displaying the environment.
This approach is best when you want your login credentials
available for the duration of an entire login session.
Store your password in an encrypted file and decrypt it
First, you have to store your password in an encrypted format.
There are several ways to do this, but
gpg is commonly
echo "my_weak_pw" | gpg -c >mysecretpw
Provide a decrypt passphrase, and you're done.
Now, you can use this by decrypting it inline as needed:
# A demo scripted use of tsshbatch with CLI password passing
MYPW=`cat mysecretpw | gpg` # User will be prompted for unlock passphrase
sshbatch.py -n joe.luser -p $MYPW hostlist1 command1 arg
sshbatch.py -n joe.luser -p $MYPW hostlist2 command2 arg
sshbatch.py -n joe.luser -p $MYPW hostlist3 command3 arg
This approach is best when you want your login credentials
available for the duration of the execution of a script. It
does require the user to type in a passphrase to unlock the
encrypted password file, but your plain text password never
appears in the wild.
Authentication Using Key Exchange
For most applications of
tsshbatch, it is much simpler to use
key-based authentication. For this to work, you must first have
pushed ssh keys to all your hosts. You then instruct
to use key-based authentication rather than name and password. Not
only does this eliminate the need to constantly provide name and
password, it also eliminates passing a plain text password on the
command line and is thus far more secure. This also overcomes the
problem of having different name/password credentials on different
tsshbatch will prompt for name and password
if they are not provided on the command line. To force key-based
authentication, use the
tsshbatch.py -k AIX-prod-hosts ls -al
This is so common that you may want to set it in your
$TSSHBATCH environment variable so that keys are used by
default. If you do this, there may still be times when you want
for force prompting for passwords rather than using keys. You can
do this with the
-K option which effectively overrides any
tsshbatch is smart enough to handle commands that begin with
sudo command. It knows that such commands require a
password no matter how you initially authenticate to get into the
system. If you provide a password - either via interactive entry
-p option - by default,
tsshbatch will use that same
If you provide no password - you're using
-k and have not
provided a password via
tsshbatch will prompt you for
sudo should use.
You can force
tsshbatch to ask you for a
sudo password with
-S option. This allows you to have one password for
initial login, and a different one for
Any time you a prompted for a
sudo password and a login
password has been provided (interactive or
-p), you can accept
this as the
sudo password by just hitting
tsshbatch makes a reasonable effort to scan your
command line and/or command file contents to spot
explicit invocations of the form
sudo .... It will
ignore these if they are inside single- or double quoted
strings, on the assumption that you're quoting the
sudo ... for some other purpose.
However, this is not perfect because it is not a full
reimplementation of the shell quoting and aliasing
features. For example, if you invoke an alias on the
remote machine that resolves to a
sudo command, or
you run a script with a
sudo command in it,
tsshbatch has no way to determine what you're trying
to do. For complex applications, it's best to write a
true shell script, push it all the machines in question
-P, and then have
tsshbatch remotely invoke
sudo myscript or something similar.
As always, the best way to figure out what the program
thinks you're asking for is to run it in test mode
and look at the diagnostic output.
Precedence Of Authentication Options
tsshbatch supports these various authentication options
in a particular heirarchy using a "first match wins"
scheme. From highest to lowest, the precedence is:
- Key exchange
- Forced prompting for name via -N. Notice this cancels
any previously requested key exchange authentication.
- Command Line/$TSSHBATCH environment variable sets name
- Name picked up from $USER (Default behavior)
If you try to use Key Exchange and
tsshbatch detects a command
sudo, it will prompt you for a password anyway.
This is because
sudo requires a password to promote privilege.
-P options specify file
PUT respectively. Both are followed by a quoted
file transfer specification in the form:
Note that this means the file will always be stored under
its original name in the destination directory. Renaming
isn't possible during file transfer.
tsshbatch always does
PUTs then any
outstanding command (if any) at the end of the command line. This
permits things like renaming on the remote machine after a
tsshbatch.py -P "foo ./" hostlist mv -v foo foo.has.a.new.name
GETs are a bit of a different story because you are retrieving
a file of the same name on every host. To avoid having all
but the last one clobber the previous one,
forces the files you
GET to be uniquely named by prepending
the hostname and a "-" to the actual file name:
tsshbatch.py -H myhost -G "foo ./"
This saves the file
myhost-foo in the
./ on your
These commands do not recognize any special directory shortcut
~/ like the shell interpreter might. You must name
file and directory locations using ordinary pathing conventions.
You can put as many of these requests on the command line as you
like to enable
PUTs of multiple files. You cannot,
however, use filename wildcards to specify multi-file operations.
You can put multiple
PUTs on the command line
for the same file. They do not override each other but are
cummulative. So this:
tsshbatch.py -P"foo ./" -P"foo /tmp" ...
Would put local file
foo in both
/tmp on each
host specified. Similarly, you can specify multiple files to
from remote hosts and place them in the same local directory:
tsshbatch.py -G"/etc/fstab ./tmp" -G"/etc/rc.conf ./tmp" ...
tsshbatch aborts if any file transfer fails. This
is unlike the case of failed commands which are reported but do
not abort the program. The rationale' for this is that you may
be doing both file transfer and command execution with a single
tsshbatch invocation, and the commands may depend on a file
being transfered first.
If you are sure no such problem exists, you can use the
to disable abort-after-failure semantics on file transfer. In this
case, file transfer errors will be reported, but
continue on to the next transfer request.
tsshbatch does not preserve file permissions when
transferring files. Recall that commands are always
run after file transfers, so you can manually manage
permissions like this:
tsshbatch.py -P"myfile ./tmp" hostlist chmod 640 ./tmp/myfile
This gets pretty clumsy for transferring more than one
or two files. A better way to do this is to create a
tarball of the source files,
tarball where you want it, and then untar it.
hostlistfile can be freely commented
# character. Everything from that character to the
end of that line is ignored. Similarly, you can use whitespace
freely, except in cases where it would change the syntax of a
command or host name.
You may also include other files as you wish with the
filename directive anywhere in the
hostlistfile. This is useful for breaking up long lists of
things into smaller parts. For example, suppose you have three
host lists, one for each major production areas of your network:
You might typically run different
tsshbatch jobs on each
of these sets of hosts. But suppose you now want to run a
job on all of them. Instead of copying them all into a
master file (which would be instantly obsolete if you
changed anything in one of the above files), you could
hosts-all with this content:
That way if you edited any of the underlying files, the
hosts-all would reflect the change.
Similarly you can do the same thing with the
group similar commands into separate files and include them.
tsshbatch does not enforce a limit on how deeply nested
.includes can be. An included file can include another file and
so on. However, if a circular include is detected, the program will
notify you and abort. This happens if, say, file1 includes file2,
file2 includes file3, and file3 includes file1. This would create
an infinite loop of includes if permitted. You can, of course,
include the same file multiple times, either in a single file or
throughout other included files, so long as no circular include is
tsshbatch supports the ablity to search paths to find files
you've referenced. The search path for
cmdfiles is specified
$TSSHBATCHCMDS environment variable. The
hostlistfiles search path is specified in the
$TSSHBATCHHOSTS environment variable. These are both in
standard path delimited format for your operating system. For
example, on Unix-like systems these look like this:
And so forth.
These paths are honored both for any files you specify on the
command line as well as for any files you reference in a
.include directive. This allows you to maintain libraries of
standard commands and host lists in well known locations and
.include the ones you need.
tsshbatch will always first check to see if a file you've
specified is in your local (invoking) directory and/or whether it
is a fully qualified file name before attempting to look down a
search path. If a file exist in several locations, the first
instance found "wins". So, for instance, if you have a file called
myhosts somewhere in the path defined in
you can override it by creating a file of same name in your current
tsshbatch also checks for so-called "circular includes" which
would cause an infinite inclusion loop. It will abort upon
discovering this, prior to any file transfers or commands being
tsshbatch allows you to define variables which will then be
used to replace matching strings in
hostlistfiles, and file transfer specifications. For example,
suppose you have this in a
At runtime, the program will actually connect to
host2.my.domain.com, and so on.
This allows for ease of modularization and maintenance of your
Similarly, you might want define
MYCMD=some_long_string so you
don't have to type
some_long_string over and over again in a
There are some "gotchas" to this:
The general form of a variable definition is:
.define name = value
You have to have a name but the value is optional.
FOO= simply replaces any subsequent
with nothing, effectively removing them.
= symbols to the right of the one right after
are just considered part of the variable's value.
Whitespace around the
= symbol is optional but allowed.
Variables are substituted in the order they appear:
.define LS = ls -alr
LS /etc # ls -alr /etc
.define LS = ls -1
LS /foo # ls -1 /foo
Variable names and values are case sensitive.
Variables may be defined in either
hostlistfiles but they are visible to any subsequent file
that gets read. For instance,
cmdfiles are read before any
hostlistfiles. Any variables you've defined in a
cmdfile that happen to match a string in one of your hostnames
will be substituted.
This is usually not what you want, so be careful. One way to
manage this is to use variables names that are highly unlikely
to ever show up in a hostname or command. That way your
commands and hostnames will not accidentally get substrings
replaced with variable values. For example, you might
use variable names like
Variable substitution is also performed on any host names
or commands passed on the command line.
Using The Current Hostname In Commands And File Transfers
There are times when it's convenient to be able to embed the name
of the current host in either a command or in a file transfer
specification. For example, suppose you want to use a single
tsshbatch to transfer files in a host-specific
way. You might name your files like this:
Now, all you have to do is this:
tsshbatch.py -xH "host 1 host2" -P "myfile.__HOSTNAME__ ./"
tsshbatch will substitute the name of the current
host in place of the string
__HOSTNAME__. (Note that these
are **double* underbars on each side of the string.*)
You can do this in commands (and commands within command files) as
tsshbatch.py -x hosts 'echo I am running on __HOSTNAME__'
Be careful to escape and quote things properly, especially from the
the command line, since
> are recognized by the shell
There are two forms of host name substitution possible. The first,
__HOSTNAME__ will use the name as you provided it, either as
an argument to
-H or from within a host file.
__HOSTSHORT__, will only use the portion of the name
string you provided up to the leftmost period.
So, if you specify
__HOSTNAME__ will be
replaced with that entire string, and
__HOSTSHORT__ will be replaced
Notice that, in no case does
tsshbatch do any DNS lookups to
figure this stuff out. It just manipulates the strings you provide
__HOSTSHORT__ are like any other
symbol you might have specified yourself with
.define. This means
you can override their meaning. For instance, say you're doing this:
tsshbatch.py -x myhosts echo "It is: __HOSTNAME__"
As you would expect, the program will log into that host, echo the
hostname and exit. But suppose you don't want it to echo something
else for whatever reason. You'd create a command file with this
__HOSTNAME__ = Really A Different Name
Now, when you run the command above, the output is:
It is: Really A Different Name
In other words,
.define has a higher precedence than
the preconfigured values of
Comments can go anywhere.
.include must be the first
non-whitespace text on the left end of a line. If you do this in a
foo .include bar
tsshbatch thinks you want to run the command
foo with an
.include bar. If you do it in a
the program thinks you're trying to contact a host called
.include bar. In neither case is this likely to be quite what you
had in mind. Similarly, everything to the right of the directive is
considered its argument (up to any comment character).
Whitespace is not significant at the beginning or end of a line but
it is preserved within
arguments as well as within commmand definitions.
Strictly speaking, you do not have to have whitespace after a
directive. This is recognized:
But this is strongly discouraged because it's really
hard to read.
tsshbatch writes the
stdout of the remote host(s) to
stdout on the local machine. It similarly writes remote
stderr output to the local machine's
stderr. If you wish to
stderr output, either redirect it on your local command
line or use the
-e option to turn it off entirely. If you want
everything to go to your local
stdout, use the
You must have a reasonably current version of Python 2.x installed.
It almost certainly will not work on Python 3.x because it uses the
commands module. This decision was made to make the
program as backward compatible with older versions of Python as
possible (there is way more 2.x around than there is 3.x).
If your Python installation does not install
have to install it manually, since
tsshbatch requires these
libraries as well.
tsshbatch has been run extensively from Unix-like systems (Linux,
FreeBSD) and has had no testing whatsoever on Microsoft Windows. If
you have experience using it on Windows, do please share with the
class using the email address below. While we do not officially
support this tool on Windows, if the changes needed to make it work
properly are small enough, we'd consider updating the code
BUGS AND MISFEATURES
You will not be able to run remote
sudo commands if the host in
question enables the
Defaults requiretty in its
configuration. Some overzealous InfoSec folks seem to think this is
a brilliant way to secure your system (they're wrong) and there's
tsshbatch can do about it.
sudo is presented a bad password, it ordinarily prints a
string indicating something is wrong.
tsshbatch looks for this
to let you know that you've got a problem and then terminates
further operation. This is so that you do not attempt to log in
with a bad password across all the hosts you have targeted. (Many
enterprises have policies to lock out a user ID after some small
number of failed login/access attempts.)
However, some older versions of
sudo (noted on a RHEL 4 host
sudo 1.6.7p5) do not return any feedback when presented
with a bad password. This means that
tsshbatch cannot tell the
difference between a successful
sudo and a system waiting for
you to reenter a proper password. In this situation, if you enter a
bad password, the the program will hang. Why?
thinks nothing is wrong and waits for the
sudo command to
complete. At the same time,
sudo itself is waiting for an
updated password. In this case, you have to kill
start over. This typically requires you to put the program in
`Ctrl-Z in most shells) and then killing that job
from the command line.
There is no known workaround for this problem.
COPYRIGHT AND LICENSING
tsshbatch is Copyright (c) 2011-2014 TundraWare Inc.
tsshbatch-license.txt file in the
program distribution. If you install tsshbatch on a FreeBSD
system using the 'ports' mechanism, you will also find this file in
DOCUMENT REVISION INFORMATION
$Id: tsshbatch.rst,v 1.144 2014/12/04 19:40:40 tundra Exp $
You can find the latest version of this program at: