tsshbatch - Run Commands On Batches Of Machines
tsshbatch.py [-NSehkvy -G 'file dest' -P 'file dest' -f cmdfile -n name -p pw ] -H 'host ..' | hostlistfile [command arg ... ]
tsshbatch
is a tool to enable you to issue a command to many
hosts without having to log into each one separately. When writing
scripts, this overcomes the ssh
limitation of not being able to
specify the password on the command line.
You can also use tsshbatch
to GET
and PUT
files
from- and to many hosts at once.
tsshbatch
also understands basic sudo
syntax and can be used
to access a host, sudo
a command, and then exit.
tsshbatch
thus allows you to write complex, hands-off scripts that
issue commands to many hosts without the tedium of manual login and
sudo
promotion. System administrators, especially, will find this
helpful when working in large server farms.
tsshbatch
supports a variety of options which can be specified
on either the command line or in the $TSSHBATCH
environment
variable:
-G spec GET file on host and write local dest directory.
spec
is a quoted pair of strings. The first specifies the path of the source file (on the remote machine) to copy. The second, specifies the destination directory (on the local machine):tsshbatch.py -G "/foo/bar/baz /tmp" hostlist
This copies
/foo/bar/baz
from every machine inhostlistfile
to the local/tmp/
directory. Since all the files have the same name, they would overwrite each other if copied into the same directory. So,tsshbatch
prepends the stringhostname:
to the name of each file it saves locally.-H hostlistfile List of hosts on which to run the command. This should be enclosed in quotes so that the list of hosts is handed to the -H option as a single argument:
-H 'host1 host2 host3'
-N Force interactive username dialog -P spec PUT file from local machine to remote machine destination directory.
spec
is a quoted pair of strings. The first specifies the path of the source file (on the local machine) to copy. The second, specifies the destination directory (on the remote machine):tsshbatch.py -P "/foo/bar/baz /tmp" hostlist
This copies
/foo/bar/baz
on the local machine to/tmp/
on every host inhostlist
.-S Force prompting for sudo password -e Don't report remote host stderr output -f cmdfile Read commands from a file. This file can be commented freely with the #
character. Leading- and trailing whitespace on a line are ignored.-h Print help information -k Use ssh keys instead of name/password credentials -n name Login name to use -p pw Password to use when logging in and/or doing sudo -v Print detailed program version information and exit -y Turn on 'noisy' reporting for additional detail on every line, instead of just at the top of the stdout
and ``stderr` reporting. This is helpful when you are filtering the output through something like grep that only returns matching lines and thus no context information.
If the -H
option is not selected, the item immediately following
the options is understood to be the name of the hostlistfile
.
This is a file that contains the name of each host - one per line - on
which to run the commands. This file can be commented freely with the
#
character. Leading- and trailing whitespace on a line are
ignored.
The last entry on the command line is optional and defines a command
to run. tsshbatch
will attempt to execute it on every host you've
specified either via -H
or a hostfile
:
tsshbatch.py -Hmyhost ls -al /etc
This will do a ls -al /etc
on myhost
.
Be careful when using metacharacters like &&, <<, >>, <, >
and so
on in your commands. You have to escape and quote them properly or
your local shell will interfere with them being properly conveyed to
the remote machine.
If you've specified a cmdfile
containing the commands you want run via
the -f
option, these commands will run before the command
you've defined on the command line. It is always the last command
run on each host.
tsshbatch
does all the GETs
, then all the PUTs
before
attempting to do any command processing. If no GETs
, PUTs
, or
commands have been specified, tsshbatch
will exit silently, since
"nothing to do" really isn't an error.
tsshbatch
respects the TSSHBATCH
environment variable. You
may set this variable with any options above you commonly use to avoid
having to key them in each time you run the program. For example:
export TSSHBATCH="-n jluser -p l00n3y"
This would cause all subsequent invocations of tsshbatch
to
attempt to use the login name/password credentials of jluser
and
l00n3y
respectively.
Different Ways To Specify Targeted Hostnames
There are two ways to specify the list of hosts on which you want to run the specified command:
On the command line via the
-H
option:tsshbatch.py -H 'hostA hostB' uname -a
This would run the command
uname -a
on the hostshostA
andhostB
respectively.Notice that the list of hosts must be separated by spaces but passed as a single argument. Hence we enclose them in single quotes.
Via a host list file:
tsshbatch.py myhosts df -Ph
Here,
tsshbatch
expects the filemyhosts
to contain a list of hosts, one per line, on which to run the commanddf -Ph
. As an example, if you want to target the hostslarry
,curly
andmoe
infoo.com
,myhosts
would look like this:larry.foo.com curly.foo.com moe.foo.com
This method is handy when there are standard "sets" of hosts on which you regularly work. For instance, you may wish to keep a host file list for each of your production hosts, each of your test hosts, each of your AIX hosts, and so on.
You may use the
#
comment character freely throughout a host list file to add comments or temporarily comment out a particular host line.You can even use the comment character to temporarily comment out one or most hosts in the list given to the
-H
command line argument. For example:tsshbatch.py -H "foo #bar baz" ls
This would run the
ls
command on hostsfoo
andbaz
but notbar
. This is handy if you want to use your shell's command line recall to save typing but only want to repeat the command for some of the hosts your originally Specified.
Authentication Using Name And Password
The simplest way to use tsshbatch
is to just name the hosts
can command you want to run:
tsshbatch.py linux-prod-hosts uptime
By default, tsshbatch
uses your login name found in the
$USER
environment variable when logging into other systems.
In this example, you'll be prompted only for your password which
tsshbatch
will then use to log into each of the machines named
in linux-prod-hosts
. (Notice that his assumes your name and
password are the same on each host!)
Typing in your login credentials all the time can get tedious after
awhile so tsshbatch
provides a means of providing them on the
command line:
tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime
This allows you to use tsshbatch
inside scripts for hands-free
operation.
If your login name is the same on all hosts, you can simplify this further by defining it in the environment variable:
export TSSHBATCH="-n joe.luser"
Any subsequent invocation of tsshbatch
will only require a
password to run.
HOWEVER, there is a huge downside to this - your plain text
password is exposed in your scripts, on the command line, and
possibly your command history. This is a pretty big security hole,
especially if you're an administrator with extensive privileges.
(This is why the ssh
program does not support such an option.)
For this reason, it is strongly recommended that you use the -p
option sparingly, or not at all. A better way is to push ssh keys
to every machine and use key exchange authentication as described
below.
However, there are times when you do have use an explicit password,
such as when doing sudo
invocations. It would be really nice
to use -p
and avoid having to constantly type in the password.
There are two strategies for doing this more securely than just
entering it in plain text on the command line:
Temporarily store it in the environment variable:
export TSSHBATCH="-n joe.luser -p my_weak_pw"
Do this interactively after you log in, not from a script (otherwise you'd just be storing the plain text password in a different script). The environment variable will persist as long as you're logged in and disappear when you log out.
If you use this just make sure to observe three security precautions:
- Clear your screen immediately after doing this so no one walking by can see the password you just entered.
- Configure your shell history system to ignore commands beginning with
export TSSHBATCH
. That way your plain text password will never appear in the shell command history.- Make sure you don't leave a logged in session unlocked so that other users could walk up and see your password by displaying the environment.
This approach is best when you want your login credentials available for the duration of an entire login session.
Store your password in an encrypted file and decrypt it inline.
First, you have to store your password in an encrypted format. There are several ways to do this, but
gpg
is commonly used:echo "my_weak_pw" | gpg -c >mysecretpw
Provide a decrypt passphrase, and you're done.
Now, you can use this by decrypting it inline as needed:
#!/bin/sh # A demo scripted use of tsshbatch with CLI password passing MYPW=`cat mysecretpw | gpg` # User will be prompted for unlock passphrase sshbatch.py -n joe.luser -p $MYPW hostlist1 command1 arg sshbatch.py -n joe.luser -p $MYPW hostlist2 command2 arg sshbatch.py -n joe.luser -p $MYPW hostlist3 command3 arg
This approach is best when you want your login credentials available for the duration of the execution of a script. It does require the user to type in a passphrase to unlock the encrypted password file, but your plain text password never appears in the wild.
Authentication Using Key Exchange
For most applications of tsshbatch
, it is much simpler to use
key-based authentication. For this to work, you must first have
pushed ssh keys to all your hosts. You then instruct tsshbatch
to use key-based authentication rather than name and password. Not
only does this eliminate the need to constantly provide name and
password, it also eliminates passing a plain text password on the
command line and is thus far more secure. This also overcomes the
problem of having different name/password credentials on different
hosts.
By default, tsshbatch
will prompt for name and password
if they are not provided on the command line. To force key-based
authentication, use the -k
option:
tsshbatch.py -k AIX-prod-hosts ls -al
Executing A sudo
Command
tsshbatch
is smart enough to handle commands that begin with
the sudo
command. It knows that such commands require a
password no matter how you initially authenticate to get into the
system. If you provide a password - either via interactive entry
or the -p
option - by default, tsshbatch
will use that same
password for sudo
promotion.
If you provide no password - you're using -k
and have not
provided a password via -p
- tsshbatch
will prompt you for
the password sudo
should use.
You can force tsshbatch
to ask you for a sudo
password with
the -S
option. This allows you to have one password for
initial login, and a different one for sudo
promotion.
Any time you a prompted for a sudo
password and a login
password has been provided (interactive or -p
), you can accept
this as the sudo
password by just hitting Enter
.
Precedence Of Authentication Options
tsshbatch
supports these various authentication options
in a particular heirarchy using a "first match wins"
scheme. From highest to lowest, the precedence is:
- Key exchange
- Forced prompting for name via -N
- Command Line/$TSSHBATCH environment variable sets name
- Name picked up from $USER (Default behavior)
If try to use Key Exchange and tsshbatch
detects a command
beginning with sudo
, it will prompt you for a password anyway.
This is because sudo
requires a password to promote privilege.
File Transfers
The -G
and -P
options specify file GET
and
PUT
respectively. Both are followed by a quoted
file transfer specification in the form:
"path-to-source-file path-to-destination-directory"
Note that this means the file will always be stored under its original name in the destination directory. Renaming isn't possible during file transfer.
However,
tsshbatch
always doesGETs
thenPUTs
then any outstanding command (if any) at the end of the command line. This permits things like renaming on the remote machine after aPUT
:tsshbatch.py -P "foo ./" hostlist mv -v foo foo.has.a.new.name
GETs
are a bit of a different story because you are retrieving a file of the same name on every host. To avoid having all but the last one clobber the previous one,tsshbatch
makes forces the files youGET
to be uniquely named by prepending the hostname and a ":" to the actual file name:tssbatch.py -H myhost -G "foo ./"
This saves the file
myhost:foo
in the./
on your local machine.The commands do not recognize any special directory shortcut symbols like
~/
. You must name file and directory locations using ordinary pathing conventions. You can put as many of these requests on the command line as you like to enableGETs
andPUTs
of multiple files. You cannot, however, use filename wildcards to specify multi-file operations.If any file transfer fails, for any reason, the program is aborted and no further work is done.
File Commenting And Includes
Both the cmdfile
and hostlistfile
can be freely commented
using the #
character. Everything from that character to the
end of that line is ignored. Similarly, you can use whitespace
freely, except in cases where it would change the syntax of a
command or host name.
You may also include other files as you wish with the
.include filename
directive anywhere in the cmdfile
or
hostlistfile
. This is useful for breaking up long lists
of things into smaller parts. For example, suppose you
have three host lists, one for each major production areas
of your network:
hosts-development
hosts-stage
host-production
You might typically run different tsshbatch
jobs on each
of these sets of hosts. But suppose you now want to run a
job on all of them. Instead of copying them all into a
master file (which would be instantly obsolete if you
changed anything in one of the above files), you could
create hosts-all
with this content:
.include hosts-development
.include hosts-stage
.include hosts-production
That way if you edited any of the underlying files, the
hosts-all
would reflect the change.Similarly you can do the same thing with the
cmdfile
to group similar commands into separate files and include them.
tsshbatch
does not enforce a limit on how deeply nested.includes
can be. An included file can include another file and so on. However, if a circular include is detected, the program will notify you and abort. This happens if, say, file1 includes file2, file2 includes file3, and file3 includes file1. This would create an infinite loop of includes if permitted. You can, of course, include the same file multiple times, either in a single file or throughout other included files, so long as no circular include is created.
tsshbatch
writes the stdout
of the remote host(s) to
stdout
on the local machine. It similarly writes remote
stderr
output to the local machine's stderr
. If you wish to
suppress stderr
output, either redirect it on your local command
line or use the -e
option to turn it off entirely.
You will not be able to run remote sudo
commands if the host
in question enables the Defaults requiretty
in its sudoers
configuration.
You must have a reasonably current version of Python installed. If
your Python installation does not install paramiko
you'll have to
install it manually, since tsshbatch
requires these libraries as
well.
When sudo
is presented a bad password, it ordinarily prints a
string indicating something is wrong. tsshbatch
looks for this to
let you know that you've got a problem and then terminates further
operation. This is so that you do not attempt to log in
with a bad password across all the hosts you have targeted. (Many
enterprises have policies to lock out a user ID after some small
number of failed login/access attempts.)
However, some older versions of sudo
(noted on a RHEL 4 server
running sudo
1.6.7p5) do not return any feedback when presented
with a bad password. This means that tsshbatch
cannot tell the
difference between a successful sudo
and a system waiting for you
to reenter a proper password. In this situation, if you enter a bad
password, the the program will hang. Why? tsshbatch
thinks
nothing is wrong and waits for the sudo
command to complete. At
the same time, sudo
itself is waiting for an updated password. In
this case, you have to kill tsshbatch
and start over. This
typically requires you to put the program in background (`Ctrl-Z
in most shells) and then killing that job from the command line.
There is no known workaround for this problem.
tsshbatch is Copyright (c) 2011-2013 TundraWare Inc.
For terms of use, see the tsshbatch-license.txt
file in the
program distribution. If you install tsshbatch on a FreeBSD
system using the 'ports' mechanism, you will also find this file in
/usr/local/share/doc/tsshbatch
.
Tim Daneliuk
tsshbatch@tundraware.com
$Id: tsshbatch.rst,v 1.121 2013/10/24 18:59:05 tundra Exp $
You can find the latest version of this program at:
http://www.tundraware.com/Software/tsshbatch