diff --git a/tsshbatch.rst b/tsshbatch.rst index 7fccd35..8eb9dec 100644 --- a/tsshbatch.rst +++ b/tsshbatch.rst @@ -168,7 +168,7 @@ The last entry on the command line is optional and defines a command to run. ``tsshbatch`` will attempt to execute it on every host you've -specified either via ``-H`` or a ``hostfile``:: +specified either via ``-H`` or a ``hostlistfile``:: tsshbatch.py -Hmyhost ls -al /etc @@ -215,531 +215,670 @@ FEATURES AND USE CASES ---------------------- -1) Different Ways To Specify Targeted Hostnames +The sections below describe the various features of ``tsshbatch`` in +more detail as well as common use scenarios. - There are two ways to specify the list of hosts on which you want - to run the specified command: +Different Ways To Specify Targeted Hostnames +============================================ - - On the command line via the ``-H`` option:: +There are two ways to specify the list of hosts on which you want +to run the specified command: - tsshbatch.py -H 'hostA hostB' uname -a + - On the command line via the ``-H`` option:: - This would run the command ``uname -a`` on the - hosts ``hostA`` and ``hostB`` respectively. + tsshbatch.py -H 'hostA hostB' uname -a - Notice that the list of hosts must be separated by spaces but - passed as a *single argument*. Hence we enclose them in single - quotes. + This would run the command ``uname -a`` on the + hosts ``hostA`` and ``hostB`` respectively. + + Notice that the list of hosts must be separated by spaces but + passed as a *single argument*. Hence we enclose them in single + quotes. - - Via a host list file:: + - Via a host list file:: - tsshbatch.py myhosts df -Ph + tsshbatch.py myhosts df -Ph - Here, ``tsshbatch`` expects the file ``myhosts`` to contain a - list of hosts, one per line, on which to run the command ``df - -Ph``. As an example, if you want to target the hosts ``larry``, - ``curly`` and ``moe`` in ``foo.com``, ``myhosts`` would look - like this:: + Here, ``tsshbatch`` expects the file ``myhosts`` to contain a list + of hosts, one per line, on which to run the command ``df -Ph``. As + an example, if you want to target the hosts ``larry``, ``curly`` + and ``moe`` in ``foo.com``, ``myhosts`` would look like this:: - larry.foo.com - curly.foo.com - moe.foo.com + larry.foo.com + curly.foo.com + moe.foo.com - This method is handy when there are standard "sets" of hosts on - which you regularly work. For instance, you may wish to keep a - host file list for each of your production hosts, each of your - test hosts, each of your AIX hosts, and so on. + This method is handy when there are standard "sets" of hosts on + which you regularly work. For instance, you may wish to keep a + host file list for each of your production hosts, each of your + test hosts, each of your AIX hosts, and so on. - You may use the ``#`` comment character freely throughout a - host list file to add comments or temporarily comment out a - particular host line. + You may use the ``#`` comment character freely throughout a host + list file to add comments or temporarily comment out a particular + host line. - You can even use the comment character to temporarily comment - out one or most hosts in the list given to the ``-H`` command - line argument. For example:: + You can even use the comment character to temporarily comment out + one or most hosts in the list given to the ``-H`` command line + argument. For example:: - tsshbatch.py -H "foo #bar baz" ls + tsshbatch.py -H "foo #bar baz" ls - This would run the ``ls`` command on hosts ``foo`` and ``baz`` - but not ``bar``. This is handy if you want to use your - shell's command line recall to save typing but only want to - repeat the command for some of the hosts your originally - Specified. + This would run the ``ls`` command on hosts ``foo`` and ``baz`` but + not ``bar``. This is handy if you want to use your shell's + command line recall to save typing but only want to repeat the + command for some of the hosts your originally Specified. -2) Authentication Using Name And Password +Authentication Using Name And Password +====================================== +The simplest way to use ``tsshbatch`` is to just name the hosts +can command you want to run:: - The simplest way to use ``tsshbatch`` is to just name the hosts - can command you want to run:: + tsshbatch.py linux-prod-hosts uptime - tsshbatch.py linux-prod-hosts uptime +By default, ``tsshbatch`` uses your login name found in the ``$USER`` +environment variable when logging into other systems. In this +example, you'll be prompted only for your password which ``tsshbatch`` +will then use to log into each of the machines named in +``linux-prod-hosts``. (*Notice that his assumes your name and +password are the same on each host!*) - By default, ``tsshbatch`` uses your login name found in the - ``$USER`` environment variable when logging into other systems. - In this example, you'll be prompted only for your password which - ``tsshbatch`` will then use to log into each of the machines named - in ``linux-prod-hosts``. (*Notice that his assumes your name and - password are the same on each host!*) +Typing in your login credentials all the time can get tedious after +awhile so ``tsshbatch`` provides a means of providing them on the +command line:: - Typing in your login credentials all the time can get tedious after - awhile so ``tsshbatch`` provides a means of providing them on the - command line:: + tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime - tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime +This allows you to use ``tsshbatch`` inside scripts for hands-free +operation. - This allows you to use ``tsshbatch`` inside scripts for hands-free - operation. +If your login name is the same on all hosts, you can simplify this +further by defining it in the environment variable:: - If your login name is the same on all hosts, you can simplify - this further by defining it in the environment variable:: + export TSSHBATCH="-n joe.luser" - export TSSHBATCH="-n joe.luser" +Any subsequent invocation of ``tsshbatch`` will only require a +password to run. - Any subsequent invocation of ``tsshbatch`` will only require a - password to run. +HOWEVER, there is a huge downside to this - your plain text password +is exposed in your scripts, on the command line, and possibly your +command history. This is a pretty big security hole, especially if +you're an administrator with extensive privileges. (This is why the +``ssh`` program does not support such an option.) For this reason, it +is strongly recommended that you use the ``-p`` option sparingly, or +not at all. A better way is to push ssh keys to every machine and use +key exchange authentication as described below. - HOWEVER, there is a huge downside to this - your plain text - password is exposed in your scripts, on the command line, and - possibly your command history. This is a pretty big security hole, - especially if you're an administrator with extensive privileges. - (This is why the ``ssh`` program does not support such an option.) - For this reason, it is strongly recommended that you use the ``-p`` - option sparingly, or not at all. A better way is to push ssh keys - to every machine and use key exchange authentication as described - below. +However, there are times when you do have use an explicit password, +such as when doing ``sudo`` invocations. It would be really nice to +use ``-p`` and avoid having to constantly type in the password. There +are two strategies for doing this more securely than just entering it +in plain text on the command line: - However, there are times when you do have use an explicit password, - such as when doing ``sudo`` invocations. It would be really nice - to use ``-p`` and avoid having to constantly type in the password. - There are two strategies for doing this more securely than just - entering it in plain text on the command line: + - Temporarily store it in the environment variable:: - - Temporarily store it in the environment variable:: + export TSSHBATCH="-n joe.luser -p my_weak_pw" - export TSSHBATCH="-n joe.luser -p my_weak_pw" + Do this *interactively* after you log in, not from a script + (otherwise you'd just be storing the plain text password in a + different script). The environment variable will persist as long + as you're logged in and disappear when you log out. - Do this *interactively* after you log in, not from - a script (otherwise you'd just be storing the plain text - password in a different script). The environment variable - will persist as long as you're logged in and disappear - when you log out. + If you use this just make sure to observe three security + precautions: - If you use this just make sure to observe three security - precautions: + 1) Clear your screen immediately after doing this so no one + walking by can see the password you just entered. - 1) Clear your screen immediately after doing this so no one - walking by can see the password you just entered. + 2) Configure your shell history system to ignore commands + beginning with ``export TSSHBATCH``. That way your plain + text password will never appear in the shell command history. - 2) Configure your shell history system to ignore - commands beginning with ``export TSSHBATCH``. That - way your plain text password will never appear in - the shell command history. - - 3) Make sure you don't leave a logged in session unlocked so - that other users could walk up and see your password by - displaying the environment. + 3) Make sure you don't leave a logged in session unlocked so + that other users could walk up and see your password by + displaying the environment. - This approach is best when you want your login credentials - available for the duration of an *entire login session*. + This approach is best when you want your login credentials + available for the duration of an *entire login session*. - - Store your password in an encrypted file and decrypt it - inline. + - Store your password in an encrypted file and decrypt it inline. - First, you have to store your password in an encrypted format. - There are several ways to do this, but ``gpg`` is commonly - used:: + First, you have to store your password in an encrypted format. + There are several ways to do this, but ``gpg`` is commonly used:: - echo "my_weak_pw" | gpg -c >mysecretpw + echo "my_weak_pw" | gpg -c >mysecretpw - Provide a decrypt passphrase, and you're done. + Provide a decrypt passphrase, and you're done. - Now, you can use this by decrypting it inline as needed:: + Now, you can use this by decrypting it inline as needed:: - #!/bin/sh - # A demo scripted use of tsshbatch with CLI password passing + #!/bin/sh + # A demo scripted use of tsshbatch with CLI password passing - MYPW=`cat mysecretpw | gpg` # User will be prompted for unlock passphrase + MYPW=`cat mysecretpw | gpg` # User will be prompted for unlock passphrase - sshbatch.py -n joe.luser -p $MYPW hostlist1 command1 arg - sshbatch.py -n joe.luser -p $MYPW hostlist2 command2 arg - sshbatch.py -n joe.luser -p $MYPW hostlist3 command3 arg + tsshbatch.py -n joe.luser -p $MYPW hostlist1 command1 arg + tsshbatch.py -n joe.luser -p $MYPW hostlist2 command2 arg + tsshbatch.py -n joe.luser -p $MYPW hostlist3 command3 arg - This approach is best when you want your login credentials - available for the duration of *the execution of a script*. It - does require the user to type in a passphrase to unlock the - encrypted password file, but your plain text password never - appears in the wild. + This approach is best when you want your login credentials + available for the duration of *the execution of a script*. It + does require the user to type in a passphrase to unlock the + encrypted password file, but your plain text password never + appears in the wild. -3) Authentication Using Key Exchange +Authentication Using Key Exchange +================================= - For most applications of ``tsshbatch``, it is much simpler to use - key-based authentication. For this to work, you must first have - pushed ssh keys to all your hosts. You then instruct ``tsshbatch`` - to use key-based authentication rather than name and password. Not - only does this eliminate the need to constantly provide name and - password, it also eliminates passing a plain text password on the - command line and is thus far more secure. This also overcomes the - problem of having different name/password credentials on different - hosts. +For most applications of ``tsshbatch``, it is much simpler to use +key-based authentication. For this to work, you must first have +pushed ssh keys to all your hosts. You then instruct ``tsshbatch`` to +use key-based authentication rather than name and password. Not only +does this eliminate the need to constantly provide name and password, +it also eliminates passing a plain text password on the command line +and is thus far more secure. This also overcomes the problem of +having different name/password credentials on different hosts. - By default, ``tsshbatch`` will prompt for name and password - if they are not provided on the command line. To force key-based - authentication, use the ``-k`` option:: +By default, ``tsshbatch`` will prompt for name and password if they +are not provided on the command line. To force key- authentication, +use the ``-k`` option:: - tsshbatch.py -k AIX-prod-hosts ls -al + tsshbatch.py -k AIX-prod-hosts ls -al - This is so common that you may want to set it in your - ``$TSSHBATCH`` environment variable so that keys are used by - default. If you do this, there may still be times when you want - for force prompting for passwords rather than using keys. You can - do this with the ``-K`` option which effectively overrides any - prior ``-k`` selection. +This is so common that you may want to set it in your ``$TSSHBATCH`` +environment variable so that keys are used by default. If you do +this, there may still be times when you want for force prompting for +passwords rather than using keys. You can do this with the ``-K`` +option which effectively overrides any prior ``-k`` selection. -4) Executing A ``sudo`` Command +Executing A ``sudo`` Command +============================ - ``tsshbatch`` is smart enough to handle commands that begin with - the ``sudo`` command. It knows that such commands *require* a - password no matter how you initially authenticate to get into the - system. If you provide a password - either via interactive entry - or the ``-p`` option - by default, ``tsshbatch`` will use that same - password for ``sudo`` promotion. +``tsshbatch`` is smart enough to handle commands that begin with the +``sudo`` command. It knows that such commands *require* a password no +matter how you initially authenticate to get into the system. If you +provide a password - either via interactive entry or the ``-p`` +option - by default, ``tsshbatch`` will use that same password for +``sudo`` promotion. - If you provide no password - you're using ``-k`` and have not - provided a password via ``-p`` - ``tsshbatch`` will prompt you for - the password ``sudo`` should use. +If you provide no password - you're using ``-k`` and have not provided +a password via ``-p`` - ``tsshbatch`` will prompt you for the password +``sudo`` should use. - You can force ``tsshbatch`` to ask you for a ``sudo`` password with - the ``-S`` option. This allows you to have one password for - initial login, and a different one for ``sudo`` promotion. +You can force ``tsshbatch`` to ask you for a ``sudo`` password with +the ``-S`` option. This allows you to have one password for initial +login, and a different one for ``sudo`` promotion. - Any time you a prompted for a ``sudo`` password and a login - password has been provided (interactive or ``-p``), you can accept - this as the ``sudo`` password by just hitting ``Enter``. +Any time you a prompted for a ``sudo`` password and a login password +has been provided (interactive or ``-p``), you can accept this as the +``sudo`` password by just hitting ``Enter``. - .. NOTE:: ``tsshbatch`` makes a reasonable effort to scan your - command line and/or command file contents to spot - explicit invocations of the form ``sudo ...``. It will - ignore these if they are inside single- or double quoted - strings, on the assumption that you're quoting the - literal string ``sudo ...`` for some other purpose. +.. NOTE:: ``tsshbatch`` makes a reasonable effort to scan your command + line and/or command file contents to spot explicit + invocations of the form ``sudo ...``. It will ignore these + if they are inside single- or double quoted strings, on the + assumption that you're quoting the literal string ``sudo + ...`` for some other purpose. - However, this is not perfect because it is not a full - reimplementation of the shell quoting and aliasing - features. For example, if you invoke an alias on the - remote machine that resolves to a ``sudo`` command, or - you run a script with a ``sudo`` command in it, - ``tsshbatch`` has no way to determine what you're trying - to do. For complex applications, it's best to write a - true shell script, push it all the machines in question - via ``-P``, and then have ``tsshbatch`` remotely invoke - it with ``sudo myscript`` or something similar. + However, this is not perfect because it is not a full + reimplementation of the shell quoting and aliasing features. + For example, if you invoke an alias on the remote machine + that resolves to a ``sudo`` command, or you run a script + with a ``sudo`` command in it, ``tsshbatch`` has no way to + determine what you're trying to do. For complex + applications, it's best to write a true shell script, push + it all the machines in question via ``-P``, and then have + ``tsshbatch`` remotely invoke it with ``sudo myscript`` or + something similar. - As always, the best way to figure out what the program - thinks you're asking for is to run it in test mode - and look at the diagnostic output. + As always, the best way to figure out what the program + thinks you're asking for is to run it in test mode and look + at the diagnostic output. -5) Precedence Of Authentication Options +Precedence Of Authentication Options +==================================== - ``tsshbatch`` supports these various authentication options - in a particular heirarchy using a "first match wins" - scheme. From highest to lowest, the precedence is: +``tsshbatch`` supports these various authentication options in a +particular heirarchy using a "first match wins" scheme. From highest +to lowest, the precedence is: - 1. Key exchange - 2. Forced prompting for name via -N. Notice this cancels - any previously requested key exchange authentication. - 3. Command Line/$TSSHBATCH environment variable sets name - 4. Name picked up from $USER (Default behavior) + 1. Key exchange - If you try to use Key Exchange and ``tsshbatch`` detects a command - beginning with ``sudo``, it will prompt you for a password anyway. - This is because ``sudo`` requires a password to promote privilege. + 2. Forced prompting for name via -N. Notice this cancels any + previously requested key exchange authentication. + + 3. Command Line/$TSSHBATCH environment variable sets name + + 4. Name picked up from $USER (Default behavior) + +If you try to use Key Exchange and ``tsshbatch`` detects a command +beginning with ``sudo``, it will prompt you for a password anyway. +This is because ``sudo`` requires a password to promote privilege. -6) File Transfers +File Transfers +============== - The ``-G`` and ``-P`` options specify file ``GET`` and - ``PUT`` respectively. Both are followed by a quoted - file transfer specification in the form:: +The ``-G`` and ``-P`` options specify file ``GET`` and ``PUT`` +respectively. Both are followed by a quoted file transfer +specification in the form:: - "path-to-source-file path-to-destination-directory" + "path-to-source-file path-to-destination-directory" - Note that this means the file will always be stored under - its original name in the destination directory. *Renaming - isn't possible during file transfer*. +Note that this means the file will always be stored under its original +name in the destination directory. *Renaming isn't possible during +file transfer*. - However, ``tsshbatch`` always does ``GETs`` then ``PUTs`` *then* any - outstanding command (if any) at the end of the command line. This - permits things like renaming on the remote machine after a ``PUT``:: +However, ``tsshbatch`` always does ``GETs`` then ``PUTs`` *then* any +outstanding command (if any) at the end of the command line. This +permits things like renaming on the remote machine after a ``PUT``:: - tsshbatch.py -P "foo ./" hostlist mv -v foo foo.has.a.new.name + tsshbatch.py -P "foo ./" hostlist mv -v foo foo.has.a.new.name - ``GETs`` are a bit of a different story because you are retrieving - a file of the same name on every host. To avoid having all - but the last one clobber the previous one, ``tsshbatch`` makes - forces the files you ``GET`` to be uniquely named by prepending - the hostname and a "-" to the actual file name:: +``GETs`` are a bit of a different story because you are retrieving a +file of the same name on every host. To avoid having all but the last +one clobber the previous one, ``tsshbatch`` makes forces the files you +``GET`` to be uniquely named by prepending the hostname and a "-" to +the actual file name:: - tsshbatch.py -H myhost -G "foo ./" + tsshbatch.py -H myhost -G "foo ./" - This saves the file ``myhost-foo`` in the ``./`` on your - local machine. +This saves the file ``myhost-foo`` in the ``./`` on your a local +machine. - These commands do not recognize any special directory shortcut - symbols like ``~/`` like the shell interpreter might. You must name - file and directory locations using ordinary pathing conventions. - You can put as many of these requests on the command line as you - like to enable ``GETs`` and ``PUTs`` of multiple files. You cannot, - however, use filename wildcards to specify multi-file operations. +These commands do not recognize any special directory shortcut symbols +like ``~/`` like the shell interpreter might. You must name file and +directory locations using ordinary pathing conventions. You can put +as many of these requests on the command line as you like to enable +``GETs`` and ``PUTs`` of multiple files. You cannot, however, use +filename wildcards to specify multi-file operations. - You can put multiple ``GETs`` or ``PUTs`` on the command line - for the same file. They do not override each other but are - *cummulative*. So this:: +You can put multiple ``GETs`` or ``PUTs`` on the command line for the +same file. They do not override each other but are *cummulative*. So +this:: - tsshbatch.py -P"foo ./" -P"foo /tmp" ... + tsshbatch.py -P"foo ./" -P"foo /tmp" ... - Would put local file ``foo`` in both ``./`` and ``/tmp`` on each - host specified. Similarly, you can specify multiple files to ``GET`` - from remote hosts and place them in the same local directory:: +Would put local file ``foo`` in both ``./`` and ``/tmp`` on each host +specified. Similarly, you can specify multiple files to ``GET`` from +remote hosts and place them in the same local directory:: - tsshbatch.py -G"/etc/fstab ./tmp" -G"/etc/rc.conf ./tmp" ... + tsshbatch.py -G"/etc/fstab ./tmp" -G"/etc/rc.conf ./tmp" ... - By default, ``tsshbatch`` aborts if any file transfer fails. This - is unlike the case of failed commands which are reported but do - *not* abort the program. The rationale' for this is that you may - be doing both file transfer and command execution with a single - ``tsshbatch`` invocation, and the commands may depend on a file - being transfered first. +By default, ``tsshbatch`` aborts if any file transfer fails. This is +unlike the case of failed commands which are reported but do *not* +abort the program. The rationale' for this is that you may be doing +both file transfer and command execution with a single ``tsshbatch`` +invocation, and the commands may depend on a file being transfered +first. - If you are sure no such problem exists, you can use the ``-a`` option - to disable abort-after-failure semantics on file transfer. In this - case, file transfer errors will be reported, but ``tsshbatch`` will - continue on to the next transfer request. +If you are sure no such problem exists, you can use the ``-a`` option +to disable abort-after-failure semantics on file transfer. In this +case, file transfer errors will be reported, but ``tsshbatch`` will +continue on to the next transfer request. - .. WARNING:: ``tsshbatch`` does *not* preserve file permissions when - transferring files. Recall that commands are always - run *after* file transfers, so you can manually manage - permissions like this:: +.. WARNING:: ``tsshbatch`` does *not* preserve file permissions when + transferring files. Recall that commands are always run + *after* file transfers, so you can manually manage + permissions like this:: - tsshbatch.py -P"myfile ./tmp" hostlist chmod 640 ./tmp/myfile + tsshbatch.py -P"myfile ./tmp" hostlist chmod 640 ./tmp/myfile - This gets pretty clumsy for transferring more than one - or two files. A better way to do this is to create a - tarball of the source files, ``GET`` or ``PUT`` the - tarball where you want it, and then untar it. + This gets pretty clumsy for transferring more than one or + two files. A better way to do this is to create a + tarball of the source files, ``GET`` or ``PUT`` the + tarball where you want it, and then untar it. -7) Commenting +Commenting +========== - Both the ``cmdfile`` and ``hostlistfile`` can be freely commented - using the ``#`` character. Everything from that character to the - end of that line is ignored. Similarly, you can use whitespace - freely, except in cases where it would change the syntax of a - command or host name. +Both the ``cmdfile`` and ``hostlistfile`` can be freely commented +using the ``#`` character. Everything from that character to the end +of that line is ignored. Similarly, you can use whitespace freely, +except in cases where it would change the syntax of a command or host +name. -8) Includes +Includes +======== - You may also include other files as you wish with the ``.include - filename`` directive anywhere in the ``cmdfile`` or - ``hostlistfile``. This is useful for breaking up long lists of - things into smaller parts. For example, suppose you have three - host lists, one for each major production areas of your network:: +You may also include other files as you wish with the ``.include +filename`` directive anywhere in the ``cmdfile`` or ``hostlistfile``. +This is useful for breaking up long lists of things into smaller +parts. For example, suppose you have three host lists, one for each +major production areas of your network:: + + hosts-development + hosts-stage + host-production + +You might typically run different ``tsshbatch`` jobs on each of these +sets of hosts. But suppose you now want to run a job on all of them. +Instead of copying them all into a master file (which would be +instantly obsolete if you changed anything in one of the above files), +you could create ``hosts-all`` with this content:: + + .include hosts-development + .include hosts-stage + .include hosts-production + +that way if you edited any of the underlying files, the +``hosts-all`` would reflect the change. + +Similarly you can do the same thing with the ``cmdfile`` to group +similar commands into separate files and include them. + +``tsshbatch`` does not enforce a limit on how deeply nested +``.includes`` can be. An included file can include another file and +so on. However, if a circular include is detected, the program will +notify you and abort. This happens if, say, file1 includes file2, +file2 includes file3, and file3 includes file1. This would create an +infinite loop of includes if permitted. You can, of course, include +the same file multiple times, either in a single file or throughout +other included files, so long as no circular include is created. + + +Search Paths +============ + +``tsshbatch`` supports the ablity to search paths to find files you've +referenced. The search path for ``cmdfiles`` is specified in the +``$TSSHBATCHCMDS`` environment variable. The ``hostlistfiles`` search +path is specified in the ``$TSSHBATCHHOSTS`` environment variable. +These are both in standard path delimited format for your operating +system. For example, on Unix-like systems these look like this:: + + export TSSHBATCHCMDS="/usr/local/etc/.tsshbatch/commands:/home/me/.tsshbatch/commands" + +And so forth. + +These paths are honored both for any files you specify on the command +line as well as for any files you reference in a ``.include`` +directive. This allows you to maintain libraries of standard commands +and host lists in well known locations and ``.include`` the ones you +need. + +``tsshbatch`` will always first check to see if a file you've +specified is in your local (invoking) directory and/or whether it is a +fully qualified file name before attempting to look down a search +path. If a file exist in several locations, the first instance found +"wins". So, for instance, if you have a file called ``myhosts`` +somewhere in the path defined in ``$TSSHBATCHHOSTS``, you can override +it by creating a file of same name in your current working directory. + +``tsshbatch`` also checks for so-called "circular includes" which +would cause an infinite inclusion loop. It will abort upon +discovering this, prior to any file transfers or commands being +executed. + +An Overview Of Variables +======================== + +As you become more sophisticated in your use of ``tsshbatch``, you'll +begin to see the same patterns of use over and over again. Variables +are a way for you to use "shortcuts" to reference long strings +without having to type the whole string in every time. So, for example, +instead of having to type in a command like this:: + + myfinecommand -X -Y -x because this is a really long string + +You can just define variable like this:: + + .define __MYCMD__ = myfinecommand -X -Y -x because this is a really long string + +From then on, instead of typing in that long command on the command line or in +a command file, you can just use ``__MYCMD__`` and ``tsshbatch`` will substitute +the string as you defined it whenever it encounters the variable. + +Variables can be used pretty much everwhere: + + - In ``hostlistfiles`` or in the hostnames listed with ``-H``:: + + .define __MYDOMAIN__ = stage.mydomain.com + #.define __MYDOMAIN__ = prod.mydomain.com + + host1.__MYDOMAIN__ + host2.__MYDOMAIN__ + + Now you can switch ``tsshbatch`` operation from stage to prod simply + by changing what is commented out at the beginning. + + - In file transfer specifications:: + + tsshbatch.py -xP"./fstab-__MYHOSTNAME__ ./" hostlist + tsshbatch.py -xG"/etc/__OSNAME__-release ./" hostlist + + - In ``cmdfiles``:: + + .define __SHELL__ = /usr/local/bin/bash + + __SHELL__ -c myfinescript + +.. NOTE:: A variable can have pretty much any name you like excepting + the use of metacharacters like ``<`` or ``!``. But if + you are not careful, you can cause unintended errors:: + + .define foo = Slop + + myfoodserver.foods.com + + When you run ``tsshbatch`` it will then turn the server name + into ``mySlopdserver.Slopds.com`` - probably not what you + want. + + So, it's a Really Good Idea (tm) to use some kind of naming + scheme to make variables names stand out and make them + unlikely to conflict accidentally with command- and host + strings. + + +TYPES OF VARIABLES +================== + +``tsshbatch`` has three different kinds of variables: + + - *User Defined Variables* are the kind in the example above. + You, the user, define them as you wish in a ``cmdfile`` or + ``hostlistfile``. + + - *Execution Variables* run any program or script of your choosing + (on the same machine you're running ``tsshbatch``) and assign + the results to a variable. + + - *Builtin Variables* are variables the ``tsshbatch`` itself + defines. You can override their default values by creating + a User Defined Variable of the same name. + + +WHERE AND WHEN DO VARIABLES GET PROCESSED? +========================================== + +User Defined and Execution Variables are defined in either a +``hostlistfile`` or ``cmdfile``. - hosts-development - hosts-stage - host-production +Builtin Variables are defined within ``tsshbatch`` itself unless you +override them. - You might typically run different ``tsshbatch`` jobs on each - of these sets of hosts. But suppose you now want to run a - job on all of them. Instead of copying them all into a - master file (which would be instantly obsolete if you - changed anything in one of the above files), you could - create ``hosts-all`` with this content:: +User Defined Variables are *all* read in and *then* used. If you do +something like this:: - .include hosts-development - .include hosts-stage - .include hosts-production + .define __FOO__ = firstfoo + echo __FOO__ + .define __FOO__ = secondfoo - That way if you edited any of the underlying files, the - ``hosts-all`` would reflect the change. +You'll get an output of ... ``secondfoo``! Why? Because before +``tsshbatch`` tries to run anything, it has to process all the +``cmdfiles``, ``hostlistfile``, and the command line content. So, +before we ever get around to doing an ``echo __FOO__`` on some host, +the second definition of __FOO__ has been read in ... and last +definition wins. - Similarly you can do the same thing with the ``cmdfile`` to - group similar commands into separate files and include them. +Execution Variables are like User Defined Variables. They get +processed a single time *at the time they're read in* from a +``cmdfile`` or ``hostlistfile``. - ``tsshbatch`` does not enforce a limit on how deeply nested - ``.includes`` can be. An included file can include another file and - so on. However, if a circular include is detected, the program will - notify you and abort. This happens if, say, file1 includes file2, - file2 includes file3, and file3 includes file1. This would create - an infinite loop of includes if permitted. You can, of course, - include the same file multiple times, either in a single file or - throughout other included files, so long as no circular include is - created. +Builtin Variables get evaluated *every time ``tsshbatch`` prepares to +connect to a new host* (unless you've overriden them). That way, the +most current value for them is available for use on the next host. +Keep in mind that ``tsshbatch`` isn't a programming language. It's +"variables" are simple string substitutions with "last one wins" +semantics. There is no notion of scope, for example. If you define +the same variable in, say, a ``cmdfile`` and also in the +``hostlistfile``, the latter will "win". Why? Because +``hostlistfiles`` are always read in after any ``cmdfiles``. -9) Search Paths +Finally, variable references in a definition are *ignored*. Say you +do this in a ``cmdfile``:: - ``tsshbatch`` supports the ablity to search paths to find files - you've referenced. The search path for ``cmdfiles`` is specified - in the ``$TSSHBATCHCMDS`` environment variable. The - ``hostlistfiles`` search path is specified in the - ``$TSSHBATCHHOSTS`` environment variable. These are both in - standard path delimited format for your operating system. For - example, on Unix-like systems these look like this:: + .define __CLEVER __ = __REALLYCLEVER__ + .define __REALLYCLEVER__ = Not That Smart + echo __CLEVER__ - export TSSHBATCHCMDS="/usr/local/etc/.tsshbatch/commands:/home/me/.tsshbatch/commands" +You will get this output, ``__REALLYCLEVER__``! Why? Because, the +variable references on the right side of a definition statement are +never replaced. This is a concious design choice to keep variable +definition and use as simple and obvious as possible. Allowing such +"indirect" definitions opens up a treasure trove of maintenance pain +you really want to avoid. Trust us on this one. - And so forth. - These paths are honored both for any files you specify on the - command line as well as for any files you reference in a - ``.include`` directive. This allows you to maintain libraries of - standard commands and host lists in well known locations and - ``.include`` the ones you need. +User-Defined Variables +====================== - ``tsshbatch`` will always first check to see if a file you've - specified is in your local (invoking) directory and/or whether it - is a fully qualified file name before attempting to look down a - search path. If a file exist in several locations, the first - instance found "wins". So, for instance, if you have a file called - ``myhosts`` somewhere in the path defined in ``$TSSHBATCHHOSTS``, - you can override it by creating a file of same name in your current - working directory. +``tsshbatch`` allows you to define variables which will then be used +to replace matching strings in ``cmdfiles``, ``hostlistfiles``, and +file transfer specifications. For example, suppose you have this in a +``hostlistfile``:: - ``tsshbatch`` also checks for so-called "circular includes" which - would cause an infinite inclusion loop. It will abort upon - discovering this, prior to any file transfers or commands being - executed. + .define DOMAIN=.my.own.domain.com -10) Defining Variables + host1DOMAIN + host2DOMAIN + host3DOMAIN - ``tsshbatch`` allows you to define variables which will then be - used to replace matching strings in ``cmdfiles``, - ``hostlistfiles``, and file transfer specifications. For example, - suppose you have this in a ``hostlistfile``:: +At runtime, the program will actually connect to +``host1.my.own.domain.com``, ``host2.my.domain.com``, and so on. This +allows for ease of modularization and maintenance of your files. - .define DOMAIN=.my.own.domain.com +Similarly, you might want define ``MYCMD=some_long_string`` so you +don't have to type ``some_long_string`` over and over again in a +``cmdfile``. - host1DOMAIN - host2DOMAIN - host3DOMAIN +There are some "gotchas" to this: - At runtime, the program will actually connect to - ``host1.my.own.domain.com``, ``host2.my.domain.com``, and so on. - This allows for ease of modularization and maintenance of your - files. + - The general form of a variable definition is:: - Similarly, you might want define ``MYCMD=some_long_string`` so you - don't have to type ``some_long_string`` over and over again in a - ``cmdfile``. + .define name = value - There are some "gotchas" to this: + You have to have a name but the value is optional. ``.define + FOO=`` simply replaces any subsequent ``FOO`` strings with + nothing, effectively removing them. - - The general form of a variable definition is:: + Any ``=`` symbols to the right of the one right after ``name`` are + just considered part of the variable's value. - .define name = value + Whitespace around the ``=`` symbol is optional but allowed. - You have to have a name but the value is optional. ``.define - FOO=`` simply replaces any subsequent ``FOO`` strings - with nothing, effectively removing them. + - Variables are substituted in the order they appear:: - Any ``=`` symbols to the right of the one right after ``name`` - are just considered part of the variable's value. + .define LS = ls -alr + LS /etc # ls -alr /etc + .define LS = ls -1 + LS /foo # ls -1 /foo - Whitespace around the ``=`` symbol is optional but allowed. + - Variable names and values are *case sensitive*. - - Variables are substituted in the order they appear:: + - Variables may be defined in either ``cmdfiles`` or + ``hostlistfiles`` but they are *visible to any subsequent file + that gets read*. For instance, ``cmdfiles`` are read before any + ``hostlistfiles``. Any variables you've defined in a ``cmdfile`` + that happen to match a string in one of your hostnames will be + substituted. - .define LS = ls -alr - LS /etc # ls -alr /etc - .define LS = ls -1 - LS /foo # ls -1 /foo + This is usually not what you want, so be careful. One way to + manage this is to use variables names that are highly unlikely to + ever show up in a hostname or command. That way your commands and + hostnames will not accidentally get substrings replaced with + variable values. For example, you might use variable names like + ``--MYLSCOMMAND--`` or ``__DISPLAY_VGS__``. - - Variable names and values are *case sensitive*. + - Variable substitution is also performed on any host names + or commands passed on the command line. - - Variables may be defined in either ``cmdfiles`` or - ``hostlistfiles`` but they are *visible to any subsequent file - that gets read*. For instance, ``cmdfiles`` are read before any - ``hostlistfiles``. Any variables you've defined in a - ``cmdfile`` that happen to match a string in one of your hostnames - will be substituted. - This is usually not what you want, so be careful. One way to - manage this is to use variables names that are highly unlikely - to ever show up in a hostname or command. That way your - commands and hostnames will not accidentally get substrings - replaced with variable values. For example, you might - use variable names like ``--MYLSCOMMAND--`` or - ``__DISPLAY_VGS__``. +Execution Variables +=================== - - Variable substitution is also performed on any host names - or commands passed on the command line. -11) Using The Current Hostname In Commands And File Transfers +Builtin Variables +================= - There are times when it's convenient to be able to embed the name - of the current host in either a command or in a file transfer - specification. For example, suppose you want to use a single - invocation of ``tsshbatch`` to transfer files in a host-specific - way. You might name your files like this:: +There are times when it's convenient to be able to embed the name of +the current host in either a command or in a file transfer +specification. For example, suppose you want to use a single +invocation of ``tsshbatch`` to transfer files in a host-specific way. +You might name your files like this:: - myfile.host1 - myfile.host2 + myfile.host1 + myfile.host2 - Now, all you have to do is this:: +Now, all you have to do is this:: - tsshbatch.py -xH "host 1 host2" -P "myfile.__HOSTNAME__ ./" + tsshbatch.py -xH "host 1 host2" -P "myfile.__HOSTNAME__ ./" - When run, ``tsshbatch`` will substitute the name of the current - host in place of the string ``__HOSTNAME__``. (*Note that these - are **double** underbars on each side of the string.*) +When run, ``tsshbatch`` will substitute the name of the current host +in place of the string ``__HOSTNAME__``. (*Note that these are +**double** underbars on each side of the string.*) - You can do this in commands (and commands within command files) as - well:: +You can do this in commands (and commands within command files) as +well:: - tsshbatch.py -x hosts 'echo I am running on __HOSTNAME__' + tsshbatch.py -x hosts 'echo I am running on __HOSTNAME__' - Be careful to escape and quote things properly, especially from the - the command line, since ``<`` and ``>`` are recognized by the shell - as metacharacters. +Be careful to escape and quote things properly, especially from the +the command line, since ``<`` and ``>`` are recognized by the shell as +metacharacters. - There are two forms of host name substitution possible. The first, - ``__HOSTNAME__`` will use the name *as you provided it*, either as - an argument to ``-H`` or from within a host file. +There are two forms of host name substitution possible. The first, +``__HOSTNAME__`` will use the name *as you provided it*, either as an +argument to ``-H`` or from within a host file. - The second, ``__HOSTSHORT__``, will only use the portion of the name - string you provided up to the leftmost period. +The second, ``__HOSTSHORT__``, will only use the portion of the name +string you provided up to the leftmost period. - So, if you specify ``myhost1.frumious.edu``, ``__HOSTNAME__`` will be - replaced with that entire string, and ``__HOSTSHORT__`` will be replaced - by just ``myhost1``. +So, if you specify ``myhost1.frumious.edu``, ``__HOSTNAME__`` will be +replaced with that entire string, and ``__HOSTSHORT__`` will be +replaced by just ``myhost1``. - Notice that, in no case does ``tsshbatch`` do any DNS lookups to - figure this stuff out. It just manipulates the strings you provide - as hostnames. +Notice that, in no case does ``tsshbatch`` do any DNS lookups to +figure this stuff out. It just manipulates the strings you provide as +hostnames. - The symbols ``__HOSTNAME__`` and ``__HOSTSHORT__`` are like any other - symbol you might have specified yourself with ``.define``. *This means - you can override their meaning*. For instance, say you're doing this:: +The symbols ``__HOSTNAME__`` and ``__HOSTSHORT__`` are like any other +symbol you might have specified yourself with ``.define``. *This +means you can override their meaning*. For instance, say you're doing +this:: - tsshbatch.py -x myhosts echo "It is: __HOSTNAME__" + tsshbatch.py -x myhosts echo "It is: __HOSTNAME__" - As you would expect, the program will log into that host, echo the - hostname and exit. But suppose you don't want it to echo something - else for whatever reason. You'd create a command file with this - entry:: +As you would expect, the program will log into that host, echo the +hostname and exit. But suppose you don't want it to echo something +else for whatever reason. You'd create a command file with this +entry:: - __HOSTNAME__ = Really A Different Name + __HOSTNAME__ = Really A Different Name - Now, when you run the command above, the output is:: +Now, when you run the command above, the output is:: - It is: Really A Different Name + It is: Really A Different Name - In other words, ``.define`` has a *higher precedence* than - the preconfigured values of ``HOSTNAME`` and ``HOSTSHORT``. +In other words, ``.define`` has a *higher precedence* than the +preconfigured values of ``HOSTNAME`` and ``HOSTSHORT``. OTHER @@ -770,15 +909,14 @@ .includesomefileofmine .definemyvar=foo - But this is *strongly* discouraged because it's really - hard to read. + But this is *strongly* discouraged because it's really hard to read. - ``tsshbatch`` writes the ``stdout`` of the remote host(s) to ``stdout`` on the local machine. It similarly writes remote ``stderr`` output to the local machine's ``stderr``. If you wish to suppress ``stderr`` output, either redirect it on your local command line or use the ``-e`` option to turn it off entirely. If you want - everything to go to your local ``stdout``, use the ``-E`` option. + everything to go to your local ``stdout``, use the ``-E`` option. - You must have a reasonably current version of Python 2.x installed. It almost certainly will not work on Python 3.x because it uses the @@ -856,7 +994,7 @@ :: - $Id: tsshbatch.rst,v 1.144 2014/12/04 19:40:40 tundra Exp $ + $Id: tsshbatch.rst,v 1.145 2014/12/04 21:42:52 tundra Exp $ You can find the latest version of this program at: