Bash Programming

Redirection

Put these at the top of your scripts to direct bash to send all standard out and standard error created after the statement  to $log_file.  One way or another the following statements will send everything created by the current script, sub-shells, etc. to $log_file.  Your code will be cleaner and easier to maintain.

Overwrite the contents of $log_file

exec > $log_file 2>&1

Append to $log_file

exec >> $log_file 2>&1

Overwrite the contents of $log_file and send output to screen using tee

exec > >(tee $log_file) 2>&1

Append to the contents of $log_file and send output to screen using tee

exec > >(tee -a $log_file) 2>&1

Number Format (see: http://stackoverflow.com/questions/9374868/number-formatting-in-bash-with-thousand-separator

Print value with a comma-delimiting the 1,000s place

$ printf "%'.3f\n" 12345678.901
12,345,678.901

Process IDs

process id of current script

$$

one script put another script in the background (using &) – pid of background process

$!

Variables

use variables for ranges

for x in $(eval echo "{$min..$max}");

better way to use variables for ranges using seq

for i in $(seq $min $max)

Return Status (see http://www.unix.com/shell-programming-scripting/92163-command-does-not-return-exit-status-due-tee.html

When using something like

script.sh 2>&1 | tee -a $FILE

Since pipe is used to pipe to tee checking $? for the return will not give you the exit status of the script. To get the exit status use PIPESTATUS Example:

ret=${PIPESTATUS: -1}

Functions

When writing functions getting a return status may not be as simple as using echo (see: ttp://www.linuxjournal.com/content/return-values-bash-functions).

Write your function so that it accepts a variable name as part of its command line and then set that variable to the result of the function:

  function myfunc()
  {
    local  __resultvar=$1
    local  myresult='some value'
    eval $__resultvar="'$myresult'"
  }

  myfunc result
  echo $result

For more flexibility, you may want to write your functions so that they combine both result variables and command substitution:

  function myfunc()
  {
      local  __resultvar=$1
      local  myresult='some value'
      if [[ "$__resultvar" ]]; then
          eval $__resultvar="'$myresult'"
      else
          echo "$myresult"
      fi
  }

  myfunc result
  echo $result
  result2=$(myfunc)
  echo $result2

Here, if no variable name is passed to the function, the value is output to the standard output.

History

To access the last element from the previous command line use $! Example:

$ ls -l /tmp/data.txt
-rw-r--r--  1 dan.brown  wheel  95 Nov 26 22:39 /tmp/data.txt
$ rm !$
rm /tmp/data.txt
$ ls -l /tmp/data.txt
ls: /tmp/data.txt: No such file or directory

To access the entire previous line, use !! Example:

$ which ls
/bin/ls
$ ls -l `!!`
ls -l `which ls`
-rwxr-xr-x  1 root  wheel  34736 Oct 31  2013 /bin/ls

To access other elements of the previous line, use “word designators” separated by a colon (zero-based).

$ echo a b c d e
a b c d e
$ echo !!:2
echo b
b

Ranges can also be accessed:

  $ echo a b c d e
  a b c d e
  $ echo !!:3-4
  c d

There are various shortcuts, such as, ‘:$’ to refer to the last argument, ‘:^’ to refer to the first argument, ‘:*’ to refer to all the arguments (synonym to ‘:1-$’), and others. See the cheat sheet for a complete list.

Modifiers can be used to modify the behavior of a word designators (see: http://www.catonmat.net/blog/the-definitive-guide-to-bash-command-line-history/. For example:

  $ tar -xvzf software-1.0.tgz
  software-1.0/file
  ...
  $ cd !!:$:r
  software-1.0$

Here the “r” modifier was applied to a word designator which picked the last argument from the previous command line. The ‘r’ modifier removed the trailing suffix ‘.tgz’.

The “h” modifier removes the trailing pathname component, leaving the head:

  $ echo /usr/local/apache
  /usr/local/apache
  $ echo !!:$:h
  /usr/local

The “e” modifier removes all but the trailing suffix:

  $ ls -la /usr/src/software-4.2.messy-Extension
  ...
  $ echo /usr/src/*!!:$:e
  /usr/src/*.messy-Extension    # ls could have been used instead of echo

Another interesting modifier is the substitute ‘:s/old/new/’ modifier which substitutes new for old. It can be used in conjunction with ‘g’ modifier to do global substitution. For example,

  $ ls /urs/local/software-4.2 /urs/local/software-4.3
  /usr/bin/ls: /urs/local/software-4.2: No such file or directory
  /usr/bin/ls: /urs/local/software-4.3: No such file or directory
  $ !!:gs/urs/usr/
ls /usr/local/software-4.2 /usr/local/software-4.3
ls: /usr/local/software-4.2: No such file or directory
ls: /usr/local/software-4.3: No such file or directory

This example replaces all occurances of ‘urs’ to ‘usr’ and makes the command correct.

There are a few other modifiers, such as ‘p’ modifier which prints the resulting command after history expansion but does not execute it. See the cheat sheet for all of the modifiers.

Remote Copying

Copy from hdfs to local laptop

hadoop fs -cat /path/dir/file.txt | ssh dan.brown@my-machine 'cat > /tmp/file.txt'

copy from hdfs to hdfs on another machine

hadoop fs -cat /path/dir/file.txt | ssh dan.brown@my-machine 'hadoop fs -put - /data/file.txt'