(20)

Use case with a pattern:

case $var in
/*) echo "starts with /" ;;
Works in all shells, and uses no extra processes

Use `cut`:

if [ "`echo $var | cut -c1`" = "/" ] ; then .
Works in all shells, but inefficiently uses a pipe and external process for a trivial task.

Use POSIX variable truncation:

if [ "${var%${var#?}}" = "/" ]; then
Works with ksh, bash and other POSIX-compliant shells. Not obvious if you have not seen this one before. Fails on old Bourne shells. Dave Taylor in "Wicked Cool Shell Scripts" likes this one.

Use POSIX pattern match inside of [[...]]:

if [[ $var = /* ]]; then
Works with ksh, bash and other POSIX-compliant shells. Note that you must use [[...]] and no quotes around the pattern.

The [[...]] syntax is handled internally by the shell and can therefore interpret "wildcard" patterns differently than an external command. An unquoted wildcard is interpreted as a pattern to be matched, while a quoted wildcard is taken literally. The [...] syntax, even if handled internally, is treated as though it were external for backward compatability. This requires that wildcard patterns be expanded to matching filenames.

Use ksh (93 and later) and bash variable substrings:

if [ "${var:0:1}" = "/" ]; then
ksh93 and later versions, and bash, have a syntax for directly extracting substrings by character position. ${varname:start:length}

Example: ex17 display, text

(21)

eval args

The args are read as input to the shell and the resulting command executed. Allows "double" expansion of some constructs. For example, constructing a variable name out of pieces, and then obtaining the value of that variable.

netdev=NETDEV_ 
NETDEV_1=hme0         # As part of an initialization step defining multiple devices

devnum=1              # As part of a loop over those devices
ifname=$netdev$devnum # construct a variable name NETDEV_1
eval device=\$$ifname # evaluate it - device is set to hme0

exec command args

The command is executed in place of the current shell. There is no return from an exec. I/O redirection may be used. This is also used to change the I/O for the current shell.

:

The line is variable-expanded, but otherwise treated as a comment. Sometimes used as a synonym for "true" in a loop.

while :; do
  # this loop will go forever until broken by 
  # a conditional test inside, or a signal
done

unset var ...

Remove the named variables. This is not the same as setting their values to null.

typeset [+/- options] [ name[=value] ] ... (ksh only, bash uses declare for similar functions)

Set attributes and values for shell variables and functions. When used inside a function, a local variable is created. Some of the options are:

-L[n]

Left justify and remove leading blanks. The variable always has length n if specified.

-R[n]

Right justify and fill with leading blanks. The variable always has length n if specified.

-l

The named variable is always treated as an integer. This makes arithmetic faster. The reserved word integer is an alias for typeset -i.

-Z[n]

As for -R, but fill with zeroes if the value is a number

-i

Lower-case convert the named variables

-u

Upper-case convert the named variables

-r

Mark the variables as readonly

-x

Export the named variables to the enviroment

-ft

The variables are taken as function names. Turn on execution tracing.

(22)

Text variables

${#var}

returns the length of $var in characters

${var%pattern}

removes the shortest suffix of $var patching pattern

${var%%pattern}

removes the longest suffix of $var patching pattern

${var#pattern}

removes the shortest prefix of $var patching pattern

${var##pattern}

removes the longest prefix of $var patching pattern

Numeric variables

$(( integer expression ))

The $(( ... )) construction interprets the contents as an arithmetic expression (integer only). Variables are referenced by name without the "$". Most of the arithmetic syntax of the 'C' language is supported, including bit manipulations (*,/,+,-,|,&,<<,>>. Use parentheses for changing precedence).

filename=${datapath##*/}

filename is set to "datafile.dat" since the longest prefix pattern matching "*/" is the leading directory path (compare basename)

path=${datapath%/*}

path is set to "/data/public/project/trials/set1" since the shortest suffix pattern matching "/*" is the filename in the last directory (compare dirname)

i=$((i+1))

often used in while loops

(23)

Defining functions

identifier() { list; }

POSIX syntax for shell functions. Such functions do not restrict scope of variables or signal traps. The identifier follows the rules for variable names, but uses a separate namespace.

function identifier { list; }

Ksh and bash optional syntax for defining a function. These functions may define local variables and local signal traps and so can more easily avoid side effects and be reused by multiple scripts.

A function may read or modify any shell variable that exists in the calling script. Such variables are global.

(ksh and bash only) Functions may also declare local variables in the function using typeset or declare. Local variables are visible to the current function and any functions called by it.

return [n], exit [n]

Return from a function with the given value, or exit the whole script with the given value.

Example:

die()
{
   # Print an error message and exit with given status
   # call as: die status "message" ["message" ...]
   exitstat=$1; shift
   for i in "$@"; do
      print -R "$i"
   done
   exit $exitstat
}

Calling functions.

Example:

[ -w $filename ] || \
  die 1 "$file not writeable" "check permissions"

Example: Backgrounded function call. ex12 display, text

   1: #!/bin/sh
   2: 
   3: background()
   4: {
   5:    sleep 10
   6:    echo "Background"
   7:    sleep 10
   8:    # Function will return here - if backgrounded, the subprocess will exit.
   9: }
  10: 
  11: echo "ps before background function"
  12: ps
  13: background &
  14: echo "My PID=$$"
  15: echo "Background function PID=$!"
  16: echo "ps after background function"
  17: ps
  18: exit 0

Example:

vprint()
{
   # Print or not depending on global "$verbosity"
   # Change the verbosity with a single variable.
   # Arg. 1 is the level for this message.
   level=$1; shift
   if [[ $level -le $verbosity ]]; then
      print -R $*
   fi
}

verbosity=2
vprint 1 This message will appear
vprint 3 This only appears if verbosity is 3 or higher

Reuseable functions

Functions may generate output to stdout, stderr, or any other file or filehandle. Messages to stdout may be captured by command substitution (`myfunction`, which provides another way for a function to return information to the calling script. Beware of side-effects (and reducing reusability) in functions which perform I/O.

(24)

Redirecting for the whole script

exec > outfile < infile

with no command, the exec just reassigns the I/O of the current shell.

exec n>outfile

The form n<, n> opens file descriptor n instead of the default stdin/stdout. This can then be used with read -u or print -u.

Explicitly opening or duplicating file descriptors

>&n

standard output is moved to whatever file descriptor n is currently pointing to

<&n

standard input is moved to whatever file descriptor n is currently pointing to

n>file

file descriptor n is opened for writing on the named file.

n>&1

file descriptor n is set to whatever file descriptor 1 is currently pointing to.

echo "Error: program failed" >&2 

Input and output to open file descriptors (ksh)

print -u n args

print to file descriptor n.

-p

write to the pipe to a coprocess (opened by |&)

Reading from file descriptors other than stdin:

read -u n var1 var2 rest

read a line from file descriptor n, parsing by $IFS, and placing the words into the named variables. Any left over words all go into the last variable.

-p

read from the pipe to a coprocess (opened by |&)

Closing file handles

<&-

standard input is explicitly closed

>&-

standard output is explicitly closed

I/O redirection operators are evaluated left-to-right. This makes a difference in a statement like: ">filename 2>&1". (Many books with example scripts get this wrong)

"Here" documents

<< [-]string

redirect input to the temporary file formed by everything up the matching string at the start of a line. Allows for placing file content inline in a script.

   1: #!/bin/sh
   2: echo "Example of unquoted here document, with variable and command substitution"
   3: 
   4: cat <<EOF
   5:  This text will be fed to the "cat" program as 
   6:  standard input.  It will also have variable
   7:  and command substitutions performed.
   8:  I am logged in as $USER and today is `date`
   9: EOF
  10: echo
  11: echo "Example of quoted here document, with no variable or command substitution"
  12: # The terminating string must be at the start of a line.
  13: cat <<"EndOfInput"
  14:  This text will be fed to the "cat" program as standard
  15:  input.  Since the text string marking the end was quoted, it does not get 
  16:  variable and command subsitutions.
  17:  I am logged in as $USER and today is `date`
  18: EndOfInput

Example: duplex display, text

   1: #!/bin/sh
   2: # Add in the magic postscript preface to perform
   3: # duplex printer control for Xerox docuprint.
   4: 
   5: # To have this script send the files directly to the printer, use 
   6: # a subshell to collect the output of the two 'cat' commands.
   7: 
   8: ## (
   9: cat << EOP
  10: %!PS
  11: %%BeginFeature: *Duplex DuplexTumble
  12: <</Duplex true /Tumble false>> setpagedevice
  13: %%EndFeature
  14: EOP
  15: cat "$@"
  16: ## ) | lpr

(25)

This short test script can be used to generate suitable output.
ex13: display, text

echo "This goes to stdout"
echo "This goes to stdout and has foo in the line"
echo "This goes to stderr" >&2
exit 99

Pass stderr of a command into a pipeline for further processing

exec 3>&1
./ex13.sh 2>&1 1>&3 3>&- | sed 's/stderr/STDERR/' 1>&2

We duplicate stdout to another file descriptor (3), then run the first command with stderr redirected to stdout and stdout redirected to the saved descriptor (3). The result is piped into other commands as needed. The output of the pipeline is redirected back to stderr, so that stdout and stderr of the script as a whole are what we expect.

   1: #!/bin/sh
   2: # Example 14
   3: # Take stderr from a command and pass it into a pipe
   4: # for further processing.
   5: 
   6: # Uses ex13.sh to generate some output to stderr
   7: # stdout of ex13 is processed normally
   8: 
   9: # Save a copy of original stdout
  10: exec 3>&1
  11: 
  12: # stdout from ex13.sh is directed to the original stdout (3)
  13: # stderr is passed into the pipe for further processing.
  14: # stdout from the pipe is redirected back to stderr
  15: ./ex13.sh 2>&1 1>&3 3>&-  | sed 's/stderr/STDERR/' 1>&2
  16: 
  17: # 3 is closed before running the command, just in case it cares
  18: # about inheriting open file descriptors.

Capture the exit status of a command in the middle of a pipeline

exec 3>&1
ex13stat=`((./ex13.sh; echo $? >&4) | grep 'foo' 1>&3) 4>&1`

This script uses nested subshells captured in backtics. Again we first duplicate stdout to another file descriptor (3). The inner subshell runs the first command, then writes the exit status to fd 4. The outer subshell redirects 4 to stdout so that it is captured by the backtics. Standard output from the first command (inner subshell) is passed into the pipeline as normal, but the final output of the pipeline is redirected to 3 so that it appears on the original stdout and is not captured by the backtics.

If any of the commands really care about inheriting open file descriptors that they don't need then a more correct command line closes the descriptors before running the commands.

   1: #!/bin/sh
   2: # Example 15
   3: 
   4: # Uses ex13.sh to generate some output and give us an 
   5: # exit status to capture.
   6: 
   7: # Get the exit status of ex13 into $ex13stat.  
   8: # stdout of ex13 is processed normally
   9: 
  10: # Save a copy of stdout
  11: exec 3>&1
  12: # Run a subshell, with 4 duplicated to 1 so we get it in stdout.  
  13: # Capture the output in ``
  14: # ex13stat=`( ...  ) 4>&1`
  15: # Inside the subshell, run another subshell to execute ex13, 
  16: # and echo the status code to 4
  17: # (./ex13.sh; echo $? >&4)
  18: # stdout from the inner subshell is processed normally, but the 
  19: # subsequent output must be directed to 3 so it goes to the 
  20: # original stdout and not be captured by the ``
  21: ex13stat=`((./ex13.sh; echo $? >&4) | grep 'foo' 1>&3) 4>&1`
  22: 
  23: echo Last command status=$?
  24: echo ex13stat=$ex13stat
  25: 
  26: # If any of the commands really care about inheriting open file 
  27: # descriptors that they don't need then a more correct command line 
  28: # closes the descriptors before running the commands
  29: exec 3>&1
  30: ex13stat=`((./ex13.sh 3>&- 4>&- ; echo $? >&4) | \
  31:    grep 'foo'  1>&3 3>&- 4>&- ) 4>&1`
  32: echo Last command status=$?
  33: echo ex13stat=$ex13stat

Combine the above two techniques:

Example: ex16 display, text

exec 3>&1
ex13stat=`((./ex13.sh 2>&1 1>&3 3>&- 4>&- ; echo $? >&4) | \
sed s/err/ERR/ 1>&2 3>&- 4>&- ) 4>&1`

   1: #!/bin/sh
   2: # Example 16
   3: 
   4: # Uses ex13.sh to generate some output and give us an 
   5: # exit status to capture.
   6: 
   7: # Get the exit status of ex13 into ex13stat. 
   8: # stderr of ex13 is processed by the pipe, stdout
   9: # is left alone.
  10: 
  11: # Save a copy of stdout
  12: exec 3>&1
  13: 
  14: # Run a subshell, with 4 copied to 1 so we get it in stdout.  
  15: # Capture the output in backtics`
  16: # ex13stat=`(    ) 4>&1`
  17: 
  18: # In the subshell, run another subshell to execute ex13, and 
  19: # echo the status code to 4
  20: # (./ex13.sh; echo $? >&4)
  21: 
  22: # stdout from the inner subshell is directed to the original stdout (3)
  23: # stderr is passed into the pipe for further processing.
  24: # stdout from the pipe is redirected back to stderr
  25: 
  26: # Close the extra descriptors before running the commands
  27: exec 3>&1
  28: ex13stat=`((./ex13.sh 2>&1 1>&3 3>&- 4>&- ; echo $? >&4) | \
  29:   sed s/err/ERR/ 1>&2 3>&- 4>&- ) 4>&1`
  30: 
  31: echo Last command status=$?
  32: echo ex13stat=$ex13stat
  33: 

(26)

command &

Start command as a background process. Control returns immediately to the shell.

bgpid=$!

The special variable $! contains the process ID of the last background job that was started. You can save that and examine the process later (ps -p $bgpid) or send it a signal (kill -HUP $bgpid).

ksh coprocesses

command |&

Start a coprocess with a 2-way pipe to it

read -p var

Read from the pipe to the coprocess, instead of standard input

print -p args

Write to the pipe connected to the coprocess, instead of standard output

exec <&p

The input from the coprocess is moved to standard input

exec >&p

The output from the coprocess is moved to standard output

Example: ex9 display, text
A script wants to save a copy of all output in a file, but also wants a copy to the screen. This is equivalent to always running the script as
script | tee outfile

   1: #!/bin/ksh
   2: 
   3: # If we have not redirected standard output, save a copy of
   4: # the output of this script into a file, but still send a
   5: # copy to the screen.
   6: 
   7: if [[ -t 1 ]] ; then
   8:   # Only do this if fd 1 (stdout) is still connected
   9:   # to a terminal
  10: 
  11:   # We want the standard output of the "tee" process
  12:   # to go explicitly to the screen (/dev/tty)
  13:   # and the second copy goes into a logfile named $0.out
  14: 
  15:   tee $0.out >/dev/tty |&
  16: 
  17:   # Our stdout all goes into this coprocess
  18:   exec 1>&p
  19: fi
  20: 
  21: # Now generate some output
  22: print "User activity snapshot on $(hostname) at $(date)"
  23: print
  24: who

Example: ex10 display, text
Start a coprocess to look up usernames in some database. It is faster to run a single process than to run a separate lookup for each user.

   1: #!/bin/ksh
   2: # This example uses a locally written tool for Dartmouth Name Directory lookups
   3: 
   4: # Start the dndlookup program as a coprocess
   5: # Tell it to output only the canonical full name, and to not print multiple matches
   6: dndlookup -fname -u |&
   7: 
   8: # move the input/output streams so we 
   9: # can use other coprocesses too
  10: exec 4>&p
  11: exec 5<&p
  12: 
  13: echo "Name file contents:"
  14: cat namefile
  15: echo
  16: 
  17: # read the names from a file "namefile"
  18: while read uname; do
  19:   print -u4 $uname
  20:   read  -u5 dndname
  21:   case $dndname in
  22:   *many\ matches*)
  23:     # handle case where the name wasn't unique
  24:     print "Multiple matches to \"$uname\" in DND"
  25:     ;;
  26:   *no\ match*)
  27:     # handle case where the name wasn't found
  28:     print "No matches to \"$uname\" in DND"
  29:     ;;
  30:   *)
  31:     # we seem to have a hit - process the
  32:     # canonical named retrieved from dndlookup
  33:     print "Unique DND match: full name for \"$uname\" is \"$dndname\""
  34:     ;; 
  35:   esac
  36:   sleep 2
  37: done < namefile
  38: 
  39: # We've read all the names, but the coprocess
  40: # is still running.  Close the pipe to tell it
  41: # we have finished.
  42: exec 4>&-

(27)

ksh distinguishes between numerically indexed (small) arrays, and string indexed (associative) arrays. bash uses integers for all array indexing, but the integers need not be consecutive and unassigned array elements do not exist. Arrays must be declared before use, e,g. typeset -A myarray (ksh associative array), or typeset -a myarray (bash).

Array elements are set with the syntax: myarray[index]=value and referenced with the syntax ${myarray[index]}

This example shows use of an array indexed by IP addresses, as strings in ksh or as non-consecutive numbers in bash. It also demonstrates use of getopt for options processing

Example: getauthlogs display, text

   1: #!/bin/bash
   2: # $Header: $
   3: # First attempt at a consolidated auth log collection from kaserver
   4: # Timestamps in the raw files are NOT designed for easy sorting.
   5: #
   6: # Options:
   7: #  -i  -- translate hex IP addresses to dotted-decimal (relatively quick)
   8: #  -h  -- translate hex IP addresses to DNS names (somewhat slower - DNS lookups)
   9: #  -u user -- filter for the named user before translating addresses
  10: 
  11: hextodec()
  12: {
  13:    # convert the IP address in reverse-hex to dotted-decimal
  14:    echo $((0x${1:6:2})).$((0x${1:4:2})).$((0x${1:2:2})).$((0x${1:0:2}))
  15: }
  16: 
  17: hostlookup()
  18: {
  19:    # Convert a decimal IP to hostname - calls 'host' each time
  20:    hostname=$(host $1)
  21:    case $hostname in
  22:    *\ not\ found*)
  23:       # Just echo the address we tried to look up
  24:       echo "$1"
  25:       ;;
  26:    *)
  27:       # The result is word 5.  Lower-case it for consistency
  28:       set $hostname
  29:       echo "$5" | tr 'A-Z' 'a-z'
  30:       ;;
  31:    esac
  32: }
  33: 
  34: # Options
  35: iptranslate=0
  36: gethostnames=0
  37: filter=cat
  38: while getopts ihu: o ; do
  39:    case $o in
  40:    i) iptranslate=1 ;;
  41:    h) gethostnames=1; iptranslate=1 ;;
  42:    u) filter="grep $OPTARG" ;;
  43:    esac
  44: done
  45: shift $(($OPTIND-1))
  46: 
  47: # We could get the DB server names from 'fs checkservers', but it isn't obvious what is from our cell.  We
  48: # could also grep CellServDB.  I cop out and hard code one known DB server and get the others from it.
  49: masterserver=halley.dartmouth.edu
  50: serverlist=$(bos listhosts -server $masterserver| grep 'Host .* is ' | awk '{print $4}')
  51: 
  52: # If we want to filter usernames, it is more efficient to do it inline, before sorting, translation and hostname lookups
  53: 
  54: # Array to hold IP address/name conversions (associative array, ksh only)
  55: # ksh - use -A for associative array.  bash - use -a and numeric array
  56: typeset -a hostnames
  57: 
  58: (
  59: for dbserver in $serverlist; do
  60:    bos getlog -server $dbserver -file /usr/afs/logs/AuthLog
  61: done
  62: ) | grep -v 'Fetching log file' | $filter | sed -e 's/^... //' -e 's/  \([1-9]\) / 0\1 /' | sort --month-sort | \
  63:     sed '-e s/ \([0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]\)$/ 0\1/' |
  64:     while read line; do
  65:    if [[ $iptranslate == 1 ]] ; then
  66:       # Ugly!
  67:       # Sometimes we get a 7-digit hex code in the log - the kaserver apparently drops leading zeros.
  68:       # The second 'sed' in the pipe catches these are fixes them.
  69:       case $line in
  70:       *\ from\ [0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f])
  71:          # translate the reverse-hex address
  72:          iphex=${line##* from }
  73:          # bash version - index by numeric value only, but can be sparse array -- use the raw IP 
  74:          ipdec=$((0x$iphex))
  75:          frontpart=${line% from *}
  76:          if [[ $gethostnames == 1 ]]; then
  77:             # ksh - index on hex value as a string (iphex)
  78:             # bash - index on numeric value (ipdec)
  79:             index=$ipdec
  80:             if [[ -z "${hostnames[$index]}" ]]; then
  81:                hostnames[$index]="$(hostlookup $(hextodec $iphex))" 
  82:             fi
  83:             echo "$frontpart from ${hostnames[$index]}"
  84:          else
  85:             echo "$frontpart from $(hextodec $iphex)"
  86:          fi
  87:          ;;
  88:       *)
  89:          echo "$line"
  90:          ;;
  91:       esac
  92:    else
  93:       # No ip translation, just echo the whole line
  94:       echo "$line"
  95:    fi   
  96: done
  97: 

(28)

trap handler sig ...

handler is a command to be read (evaluated first) and executed on receipt of the specified sigs. Signals can be specified by name or number (see kill(1)) e.g. HUP, INT, QUIT, TERM. A Ctrl-C at the terminal generates a INT.

• A handler of - resets the signals to their default values
• A handler of '' (null) ignores the signals
• Special signal values are as follows:

  EXIT

  the handler is called when the function exits, or when the whole script exits. The exit signal has value 0.

  ERR (ksh)

  the handler is called when any command has a non-zero exit status

  DEBUG (ksh)

  the handler is called after each command.

Example: ex8 display, text

   1: #!/bin/bash
   2: # Try this under bash, ksh and sh
   3: 
   4: trap huphandler  HUP
   5: trap ''          QUIT
   6: trap exithandler TERM INT
   7: 
   8: huphandler()
   9: {
  10:    echo 'Received SIGHUP'
  11:    echo "continuing"
  12: }
  13: 
  14: exithandler()
  15: {
  16:    echo 'Received SIGTERM or SIGINT'
  17:    exit 1
  18: }
  19: ## Execution starts here - infinite loop until interrupted
  20: # Use ":" or "true" for infinite loop
  21: # SECONDS is built-in to bash and ksh.  It is number of seconds since script started
  22: : is like a comment, but it is evaluated for side effects and evaluates to true
  23: seconds=0
  24: while : ; do
  25: # while true; do
  26:    sleep 5
  27:    seconds=$((seconds + 5))
  28:    echo -n "$SECONDS $seconds - "
  29: done

(29)

• Don't use set-UID scripts.

  Most systems don't even allow a script to be made set-UID. It is impossible (due to inherent race conditions) to ensure that a set-uid script cannot be compromised. Use wrapper programs like sudo instead.

• Always explicitly set $PATH at the start of a script, so that you know exactly which external programs will be used.

• If possible, don't use temporary files. If they cannot be avoided, use $TMPDIR, and create files safely (e.g. mktemp).

  Often scripts will write to a fixed, or trivially generated temporary filename in /tmp. If the file already exists and you don't have permission to overwrite it, the script will fail. If you do have permission to overwrite it, you will delete the previous contents. Since /tmp is public write, another user may create files in it, or possibly fill it completely.
  Example:

  1. A link is created by an unprivileged user in /tmp: /tmp/scratch -> /vmunix
  2. A root user runs a script that blindly writes a scratch file to /tmp/scratch, and overwrites the operating system.

  Environment variable $TMPDIR is often used to indicate a preferred location for temporary files (e.g., a per-user directory). Some systems may use $TMP or $TEMP. Safe scratch files can be made by creating a new directory, owned and writeable only by you, then creating files in there.
  Example:

  (umask 077 && mkdir /tmp/tempdir.$$) || exit 1
  

  or (deluxe version)

  tmp=${TMPDIR:-/tmp}
  tmp=$tmp/tempdir.$RANDOM.$RANDOM.$RANDOM.$$
  (umask 077 && mkdir $tmp) || {
      echo "Could not create temporary directory" 1>&2
      exit 1
  }
  

  Alternatively, many systems have mktemp to safely create a temporary file and return the filename, which can be used by the script and then deleted.

• Check exit status of everything you do.

• Don't trust user input
  • contents of files
  • data piped from other programs
  • file names. Output of filename generation with wildcards, or directly from ls or find

Example:
Consider the effects of a file named "myfile;cd /;rm *" if processed, unquoted, by your script.

One possible way to protect against weirdo characters in file names:

# A function to massage a list of filenames 
# to protect weirdo characters
# e.g. find ... | protect_filenames | xargs command
#
# We are backslash-protecting the characters \'" ?*;
protect_filenames()
{
   sed -es/\\\\/\\\\\\\\/g \
       -es/\\\'/\\\\\'/g   \
       -es/\\\"/\\\\\"/g   \
       -es/\\\;/\\\\\;/g   \
       -es/\\\?/\\\\\?/g   \
       -es/\\\*/\\\\\*/g   \
       -es/\\\ /\\\\\ /g
}

If using GNU find and xargs, there is a much cleaner option to null-terminate generated pathnames.

(30)

• Preface scripts with a statement of purpose, author, date and revision notes
• Use a revision control system for complex scripts with a long lifetime
• Assume your script will have a long lifetime unless you are certain it won't
• Document any non-standard external utilities which your script needs
• Document your scripts with inline comments - you'll need them in a few months when you edit it.
• Treat standard input and output in the normal way, so that your script can be used in combination with other programs (the Unix toolkit philosophy)
• Be consistent in the format of your output, so that other programs can rely on it
• Use options to control behaviour such as verbosity of output. Overly chatty programs are very hard to combine with other utilities
• Use interactive features (prompting the user for information) very sparingly. Doing so renders the script unuseable in pipeline combinations with other programs, or in unattended operations.
• Test (a lot)

When not to use shell scripts

• If an existing tool already does what you need - use it.
• If the script is more than a few hundred lines, you are probably using the wrong tool.
• If performance is horrible, and the script is used a lot, you might want to consider another language.

(31)

Download a compressed tar file of all example scripts used in these notes.

• postprint display, text
  A wrapper script for printing a mix of text and postscript files

• checkpath display, text
  Check all the directories in the $PATH for possibly conflicting programs.

• run-with-timeout display, text
  Run a command with a timeout. Kill the command if it hasn't finished when the timeout expires.

• MailPkg display, text
  Tar, compress, split and uuendcode a set of files for mailing. (Blinn)

• Ptree (original) display, text
  
• Ptree (ksh version) display, text
  Runs "ps" to get a process listing and then reformats to show the process family hierarchies. The original example is pure Bourne shell and inefficient. The ksh version is a fairly simple translation to use ksh internal commands where possible, and avoid writing scratch files, and runs very much faster. (Blinn).

This entire tutorial was created from individual HTML pages using a content management system written as ksh scripts (heavily using sed to edit the pages), coordinated by make.

You can even write an entire web server as a shell script. This one is part of the LEAF (Linux Embedded Appliance Firewall) project. This wouldn't be suitable for much load, but handles occasional queries on static HTML and CGI scripts. (www.nisi.ab.ca/lrp/Packages/weblet.htm)

• sh-httpd display, text
  

(32)

Most of these commands will operate on a one or more named files, or will operate on a stream of data from standard input if no files are named.

Listing, copying and moving files and directories

ls *

list contents of a directory, or list details of files and directories.

mkdir; rmdir *

Make and Remove directories.

rm; cp; mv *

Remove (delete), Copy and Move (rename) files and directories

touch *

Update the last modifed timestamp on a file, to make it appear to have just been written.

If the file does not exist, a new zero-byte file is created, which is often useful to signify that an event has occurred.

tee

Make a duplicate copy of a data stream - used in pipelines to send one copy to a log file and a second copy on to another program. (Think plumbing).

Displaying text, files or parts of files

echo *

Echo the arguments to standard output -- used for messages from scripts. Some versions of "sh", and all csh/ksh/bash shells internalized "echo".

Conflicts sometimes arise over the syntax for echoing a line with no trailing CR/LF. Some use "\c" and some use option "-n". To avoid these problems, ksh also provides the "print" command for output.

cat *

Copy and concatenate files; display contents of a file

head, tail *

Display the beginning of a file, or the end of it.

cut

Extract selected fields from each line of a file. Often awk is easier to use, even though it is a more complex program.

wc

Count lines, words and characters in the input.

Compression and archiving

compress; gzip, zip; tar *

Various utilities to compress/uncompress individual files, combine multiple files into a single archive, or do both.

Sorting and searching for patterns

sort *

Sort data alphabetically or numerically.

grep *

Search a file for lines containing character patterns. The patterns can be simple fixed text, or very complex regular expressions.

The name comes from "Global Regular Expression and Print" -- a function from the Unix editors which was used frequently enough to warrant getting its own program.

uniq *

Remove duplicate lines, and generate a count of repeated lines.

wc *

Count lines, words and characters in a file.

System information (users, processes, time)

date *

Display the current date and time (flexible format). Useful for conditional execution based on time, and for timestamping output.

ps *

List the to a running processes.

kill *

Send a signal (interrupt) to a running process.

id

Print the user name and UID and group of the current user (e.g. to distinguish priviledged users before attempting to run programs which may fail with permission errors)

who

Display who is logged on the system, and from where they logged in.

uname *

Display information about the system, OS version, hardware architecture etc.

mail *

Send mail, from a file or standard input, to named recipients. Since scripts are often used to automate long-running background jobs, sending notification of completion by mail is a common trick.

logger

Place a message in the central system logging facility. Scripts can submit messages with all the facilities available to compiled programs.

hostname

Display the hostname of the current host - usful to keep track of where your programs are running

Conditional tests

test; [ *

The conditional test, used extensively in scripts, is also an external program which evaluates the expression given as an argument and returns true (0) or false (1) exit status. The name "[" is a link to the "test" program, so a line like:
if [ -w logfile ]
actually runs a program "[", with arguments "-w logfile ]", and returns a true/false value to the "if" command.

In ksh and most newer versions of sh, "[" is replaced with a compatible internal command, but the argument parsing is performed as if it were an external command. Ksh also provides the internal "[[" operator, with simplified syntax.

Stream Editing

awk *

A pattern matching and data manipulation utility, which has its own scripting language. It also duplicates much functionality from 'sed','grep','cut','wc', etc.

Complex scripts can be written entirely using awk, but it is frequently used just to extract fields from lines of a file (similar to 'cut').

sed *

Stream Editor. A flexible editor which operates by applying editing rules to every line in a data stream in turn.

Since it makes a single pass through the file, keeping only a few lines in memory at once, it can be used with infinitely large data sets. It is mostly used for global search and replace operations. It is a superset of 'tr', 'grep', and 'cut', but is more complicated to use.

tr

Transliterate - perform very simple single-character edits on a file.

Finding and comparing files

find *

Search the filesystem and find files matching certain criteria (name pattern, age, owner, size, last modified etc.)

xargs *

Apply multiple filename arguments to a named command and run it.

Xargs is often used in combination with "find" to apply some command to all the files matching certain criteria. Since "find" may result in a very large list of pathnames, using the results directly may overflow command line buffers. Xargs avoids this problem, and is much more efficient than running a command on every pathname individually.

diff *

Compare two files and list the differences between them.

basename pathname

Returns the base filename portion of the named pathname, stripping off all the directories

dirname pathname

Returns the directory portion of the named pathname, stripping off the filename

Arithmetic and String Manipulation

expr *

The "expr" command takes an numeric or text pattern expression as an argument, evaluates it, and returns a result to stdout. The original Bourne shell had no built-in arithmetic operators. E.g.
expr 2 + 1
expr 2 '*' '(' 21 + 3 ')'
Used with text strings, "expr" can match regular expressions and extract sub expressions. Similar functionality can be achived with sed. e.g.
expr SP99302L.Z00 : '[A-Z0-9]\{4\}\([0-9]\{3\}\)L\.*'

dc

Desk Calculator - an RPN calculator, using arbitrary precision arithmetic and user-specified bases. Useful for more complex arithmetic expressions than can be performed internally or using expr

bc

A preprocessor for dc which provides infix notation and a C-like syntax for expressions and functions.

Merging files

paste

Merge lines from multiple files into tab-delimited columns.

join

Perform a join (in the relational database sense) of lines in two sorted input files.

(33)

Books

• The New KornShell Command And Programming Language, by Morris I. Bolsky, David G. Korn (Contributor). More info
  
  
• Learning the Korn Shell, 2nd Edn. by Bill Rosenblatt and Arnold Robbins. More info
  
  
• Korn Shell Programming by Example, by Dennis O'Brien, David Pitts (Contributor). More info
  
  
• The Korn Shell Linux and Unix Programming Manual (2nd Edn) by Anatole Olczak. More info
  
  
• Portable Shell Programming: An Extensive Collection of Bourne Shell Examples by Bruce Blinn. More info
  Examples from this book can be downloaded for study.
  
  
• Linux Shell Scripting with Bash by Ken O. Burtch. More info
  
  
• Unix Shell Programming by Stephen Kochan and Patrick Wood (third Edition). More info
  
  
• Teach yourself Shell Programming in 24 Hours, by S. Veeraraghavan. SAMS 2nd Edn. (2002) More info
  
  
• Mastering Unix Shell Scripting by Randal K. Michael, Wiley (2003) More info Light on basics, but develops scripting through examples. Ksh only. Examples can be downloaded from the Wiley site (www.wiley.com/legacy/compbooks/michael/).
  
  
• Wicked Cool Shell Scripts by Dave Taylor, No Starch Press (2004) More info Develops scripting entirely through examples, drawn from Linux and OSX in addition to traditional Unix. Recommended, but not for beginners. Examples can be downloaded from the Intuitive site (www.intuitive.com/wicked/wicked-cool-shell-script-library.shtml).
  
  
• Unix Power Tools, by S. Powers, J. Peek, T. O'Reilly, M. Loudikes et al. More info

Online Resources

• Shelldorado (http://www.shelldorado.com)
  Lots of links to scripting resources

• Kornshell (http://www.kornshell.com)
  The official Korn shell home page, with download links.

• Mac OSX Unix tutorial (http://www.osxfaq.com/Tutorials/LearningCenter/)
  Good resource on advanced use of OSX and Unix shell scripting in general

Unix-like shells and utilities for Microsoft Windows

• U/Win (http://www.research.att.com/sw/tools/uwin/)
  A free port of ksh and Unix command line utilities, plus Windows DLL for Unix compatability. Developed by AT&T Research.

• Cygwin (http://www.cygwin.com/)
  A free Linux-like environment for Windows. Provides bash, command line utilities and DLLs. Developed by RedHat. An X server is also available.

• MKS Toolkit (http://www.mkssoftware.com/)
  A commercial ksh clone and command line utilities, plus DLL for Unix compatability. An X server is also available.

• Microsoft Services for UNIX (http://www.microsoft.com/windows/sfu/)
  A POSIX environment for Windows, with ksh, csh, command line tools, libraries and software development tools. Developed by Interix and bought by Microsoft. Free download.