Difference between revisions of "Intro to Bash Scripting"

Revision as of 18:51, 8 November 2008

Class Description

The class: a four week course on Tues evenings covering basic and intermediate scripting in the bash shell. We will examine a file with about 15 lines of code together each night, modify the code, run it, and come up with our own scripts. We will also get to know the "Advanced Bash Scripting Guide" [1] and the Gnu "Bash Reference Manual" [2] in some depth and learn to research and solve our own programming problems. User projects are encouraged -- bring your problems and we will solve them together!

I am a fifth year Ph D student from Berkeley in Demography, writing an anthropology of a small lumber town in Oregon. I have worked as a programmer in Linux for almost 10 years, and I am currently employed part time as a software project manager at Portland State.

The first class target: Oct 28, 5:00 to 6:30, 2008. I will probably have to take off a week and delay a class in the middle of the sequence.

Some helpful links to get us started:

http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html

http://tldp.org/LDP/abs/html/

http://www.gnu.org/software/bash/manual/bashref.html

http://www.intuitive.com/wicked/wicked-cool-shell-script-library.shtml

Class Outline

Day One (2008-10-28): What is a "script" and what is a "variable". We will write a simple script in nano, with comments, a "shebang" line, appropriate permissions, and simple output. We work on the idea of a variable, using shell expansion to assign output to variables, interpolating variables, and exporting environment variables. We will also examine the output and input streams ("stdin", "stdout", "stderr"). We will "comment-out" code. Finally we will talk about style, including indentation, variable names, trickiness, and comment-first scripting.

Day Two (2008-11-04): "For-loops", "word splitting", and printf. We will explore the for-loop in all its glory, going over lists stored in variables. This will require a discussion of how Bash automatically splits strings into words and how we can control this through quoting syntax. We will also make interesting formatting commands with the printf command.

Day Three (2008-11-11): Conditionals ("if/ then" statements). We will show how to write "if" and "case" statements, and incorporate pattern matching and "file tests" into our scripts.

Day Four (2008-11-25): File input, "while-read", and useful scripting. We will investigate how to get an input line from a file, parse it into useable pieces, and do interesting things with that data (like write a simple email "spambot").

Class Approach

Each class period I will have a file of code that is on the wiki. We will go over it together, showing you how to look up questions using online documentation. You will type examples from this wiki page, with modifications to make your own scripts, run them from the command line, and fix bugs.

Code Listings and Class Notes

Every day instructions

Each new class, make a directory classX, cd to it, and do your work there; then if you want to send yourself the files via email you have the option. Open two terminals and cd to this directory in both.

Each time we cover a script, open a file in your favorite editor named classX-scriptY.sh where X stands for the class (1, 2, 3, 4), and Y stands for the script we are working on in class. Type all the code in. Save the file. In your other terminal window, chmod 0700 the script, then try to run it.

Each day please also open the three links above to the various pieces of bash documentation. As we have questions we will try to look up the answers here rather than relying on me. (Teach a person to fish...)

Note that this class will only cover the tip of the iceberg for Bash scripting; to learn more -- follow the bash links above, type in the code you find while you read the explanations, and experiment.

Note also that this class is a work in progress, and I might jump into the wiki to make a change to the notes or the code for next time. Feel free to suggest changes or ask questions, especially if you buy me dinner after class.

First day -- scripts and variables

What is a shell script anyway? It is ...

... a file of text ...

... full of unix commands, variables, and control structures ...

... that usually executes from top to bottom ...

... using variables to hold data ...

... and loops and conditionals to do fancy programming stuff...

... with a way to read input and write output ...

... that probably has some "side-effects".

Why learn shell? Old-school Unix style? Class?

Script 1

Following just prints my name. Run it as is, but then change it to print your name.

#!/bin/bash
# 
# script 1 wsprague@nospam.org 2008-10-29
# Script just prints my name
#

FIRSTNAME='Webb' 
LASTNAME='S.'
echo "Hello, $FIRSTNAME $LASTNAME"

Edit it by running the following: nano class1-script1.sh, then save it (ctrl-O in nano), then chmod 0700 class1-script1.sh, and run as

./class1-script1.sh

in the other terminal window.

Script 2

This script takes a parameter from the command line and uses it as the name: "variable expansion". Note the expansion, but also how different quotes or lack thereof have different effects.

#!/bin/bash
# 
# script 2 wsprague@nospam.org 2008-10-29
# Prints names from command line
#

NAME=$1
echo "Hello, $NAME."
echo 'Hello, $NAME.'
echo Hello, $NAME.
echo Hello, "$NAME".
echo Hello, \"$NAME\".
echo "Hello, ${NAME}with text."
echo "Hello, $NAMEwith text."

try from the command line:

 
./class1-script2.sh 
./class1-script2.sh Foobar
./class1-script2.sh "Foobar Smith"
./class1-script2.sh Foobar Smith

Script 3

This script does some basic math, and then outputs it using variable expansion. Check out the quoting. Also note what happens when we try to do math on weird input.

#!/bin/bash
#
# Script adds two numbers from command line

LEFT=$1
RIGHT=$2
RES=$(( $LEFT + $RIGHT ))
echo $(( $LEFT + $RIGHT ))
echo '$(( $LEFT + $RIGHT ))'
echo "$(( $LEFT + $RIGHT ))"
echo "$LEFT + $RIGHT = $RES."
echo '$LEFT + $RIGHT = $RES.'  # Why does this do what it does?

try from the command line:

./class1-script3.sh 100 10        
./class1-script3.sh 'one hundred' 'ten' 
./class1-script3.sh 34

Script 4

This script does some "shell expansion" using the unix command "date",
   which gives a formatted string of the date; use "date --help" to see more).

#!/bin/bash
# 
# Script 4.  Script calculates the year if we give it years from now.
#

YEAR=$( date +'%Y' )
YEARS_FORWARD=$1
echo "Start at year $YEAR, finish at year $(($YEAR + $YEARS_FORWARD))"

Try from the command line

./class1-script4.sh
./class1-script4.sh 10
./class1-script4.sh "ten"

Script 5

#!/bin/bash
#
# Script 5 takes a filename, strips any spaces from it, and moves the original file
#    to the new name.

RES=$( echo $1 | sed 's/ /_/g' )
echo "After stripping of spaces, \"$1\" looks like \"$RES\""  # Why do we escape the quotes here?
echo mv "$1" "$RES" # try this same script without "echo" here, after touching a file with a name that you put into the parameters of the script

Try from the command line:

touch "a file with spaces in the name"
./class1-script5.sh "a quoted filename with spaces"
./class1-script5.sh an unquoted filename with spaces
././class1-script5.sh a_filename_wo_spaces

Discussion

Coding style -- variable names, comments, indentation, trickiness.

Comment first design.

"Commenting out" code.

Second day -- for-loops

What is a for-loop? It is a bash construct that repeatedly grabs one item from a sequence of data separated by whitespace, does something with that piece of the sequence, until there is no more data.

Here is the paradigm:

for VARIABLE in SEQUENCE; do
  # bunch of statements in here that are done repeatedly
  # referencing $VARIABLE
  # more statements
done

Finger exercises

Edit a few files with nano, find them with ls, and delete them with rm

Run the command `seq 1 10` from the command line

Run the command `printf "hello %s, my name is %s, I am %i years old" Tarzan Jane 32`

Run the command `factor 144` from the command line

Script 0

#!/bin/bash
# 
# script 0 wsprague@nospam.org 2008-10-29
# Basic for loop with seq and printf
#

START=$1
FINISH=$2
FULL_SEQ=$( seq $START $FINISH ) 
COUNT=0

printf "Start = %i,  finish = %i\n" $START  $FINISH   
for X in $FULL_SEQ; do
    echo "touching file.$X"
    touch "file.$X"
    COUNT=$(( $COUNT+1 ))
done
printf "finished working on %i files\n" $COUNT

Save this as class2.script0.sh, then chmod 0700 class2.script0.sh. Run as follows:

./class2.script0.sh 1 10
./class2.script0.sh 4 14

Examine your directory to see the new files and their names. Those are from this script.

Script 1

#!/bin/bash                                                                           
#                                                                                     
# script 1 wsprague@nospam.org 2008-10-29                                             
# Do some fancy formatting with printf,                                               
#    by calculating the first 10 "orders of magnitude", and printing them             

MAX=7
SEQ=$( seq 0 $MAX)
echo $SEQ
for I in $SEQ; do
    OUT=$(( 10 ** $I ))
    printf "The %3i order of magnitude = %i.\n" $I $OUT
done

Run this as

./class2.script2.sh

Modify it to make "MAX" store the number from $1.

Script 2

The following factors a number and creates a bunch of files.

#!/bin/bash                                                                                 
#                                                                                           
# script 2 wsprague@nospam.org 2008-10-29                                                   
# Takes a name and a number as parameters,                                                  
#    factors the number, touches all files "$name.$number".                                 

NAME=foobar # use $1 
NUMBER=12   factor $NUMBER | sed 's/^[0-9]*://g' )
printf "Factors working on: %s\n" "$FACTORS"
echo

TEST=1
COUNT=0
for SUFFIX in $FACTORS; do
    TEST=$(($TEST * $SUFFIX))
    COUNT=$(($COUNT + 1))
    touch "$NAME.$SUFFIX"
    printf "Touched:  prefix = %s, suffix = %i\n" $NAME $SUFFIX
done
printf "\nFinished working on %i files\n" $COUNT

Run this script as

./class2.script2.sh

Then modify it to set "NAME" and "NUMBER" from the command line.

                                                                       
#!/bin/bash                                                                                                       
#                                                                                                                 
# script 3 wsprague@nospam.org 2008-10-29                                                                         
# Generate times tables by using a nested loop.                                                                   

START=$1
FINISH=$2
INCREMENT=1  # try changing this                                                                                  
SEQ=$(seq $START $FINISH )

# Print top row                                                                                                   
printf "      "
for x in $SEQ; do
    printf "%4i " $x
done

# Print top row separator                                                                                         
printf "\n     "
for x in $SEQ; do
    printf "_____"
done

echo

# Fill in each row with left label and cell result                                                                
for x in $SEQ; do
    printf "\n%4i| " $x
    for y in $SEQ; do
        printf "%4i " $(( $x * $y ))
    done;
done;
printf "\n"

Run this as

 
./class2-script3.sh 1 12

Discussion

Debugging loops with echo statements

Precalculating things like sequences

Code style -- when to add whitespace between sections and stanzas

Third day -- conditionals

http://www.gnu.org/software/bash/manual/bashref.html#Conditional-Constructs

What is "truth"?

Script 1 -- determine relative order of two numbers, and check input validity

#!/bin/bash                                                                                                       
#                                                                                                                 
# script 1 wsprague@nospam.org 2008-10-29                                                                         
# Categorize numbers based on input.  
#    Also check input (finally!).                                                                                       

USAGE='./script1.sh num1 num2'
ERROR="Error.  Usage: $USAGE"

if [[ $1 ]]; then                 # $1 is "true" if something is in there                                       
    #echo "Thanks for the shell variable" 1>&2 # note "redirection" of stdout to stderr                         
    LEFT=$1
    shift
else
    echo $ERROR
    exit 1
fi

if [[ $1 ]]; then
    #echo "Thanks for the shell variable" 1>&2                                                                  
    RIGHT=$1
    shift
else
    echo $ERROR
    exit 1
fi

if [[ $LEFT < $RIGHT ]]; then
    echo "$LEFT is strictly lesser than $RIGHT _lexicographically_"
elif [[ $LEFT < $RIGHT ]]; then
    echo "$LEFT is strictly greater than $RIGHT _lexicographically_"
else
    echo "$LEFT is strictly greater than $RIGHT _lexicographically_"
fi

Do the chmod 0700 etc dance. Try this as

./script1 aardvark zebra
./script1 1 9
./script1 10 9
./script1 foo

Script 2 -- filter prime numbers

Prints out prime numbers, uses a lot of crazy conditional operations.

#!/bin/bash
#  script2.sh
#  displays prime numbers less than input parameter.=
#  tests for input validity
#  breaks out of loop once go over max
#  does lots of stuff!

# write a function that tweaks factor command
function myfactor (){
    OUT=$(gfactor $1 | sed 's/^[0-9]*: //')
    echo "$OUT"
}

# Verify input
USAGE="script2.sh num"
if [[ $1 && $(echo $1 | grep '^[0-9][0-9]*$') ]]; then             ## Note boolean operator
    MAX=$1; 
    shift
else
    echo $USAGE
    exit 1;                     ## "1" signifies error.  Run a script and then try "echo $?" 
fi

# Verify script has not already been run for given number
if [[ -e $MAX ]]; then
    echo "Already ran script for number $MAX.  rm file to do repeat." 1>&2
    exit 1
else
    touch $MAX
fi

# Get prime numbers, but not above MAXMAX
MAXMAX=100
for X in $(gseq $MAX); do 
    
    # Test whether have more factors than input number
    FACTORS=$( myfactor $X ) 
    if [[ $X == $FACTORS  ]]; then
        echo $X
    else
        echo "foobar" > /dev/null                       # do nothing
    fi

    # test whether above MAXMAX, "break" out of loop if so.
    if [[ $X -gt $MAXMAX ]] ; then
        echo "won't let $X go above $MAXMAX. Breaking out of loop"
        break
    fi

done

# do something to highlight how the break statement works
echo bye

Run this as

./script2.sh 
./script2.sh 80
./script2.sh 800
./script2.sh blahblahlbah

Script 3 -- using case

#!/bin/bash
#
# Script that does some silliness  case statements

USAGE="./$0 input" 
if [[ $1 ]]; then
    ARG=$1; shift
else
    echo "$USAGE"
    exit
fi

case $ARG in 
    foo | bar | baz) echo "speaking unix baby-talk";;
    mama | dada) 
        echo "speaking real baby-talk";; ## note multiline block
    [0-9]*) echo "numbers"
        touch $ARG.file
        ;;
    mama ) echo ma-ma;;         ## we won't ever do this
    *) echo other stuff;;
esac

Run this as

./script3.sh 
./script3.sh 12
./script3.sh mama
./script3.sh dada
./script3.sh "this is a long line"

Discussion

...

Fourth day -- while-read and functions

We are going to read a file with email addresses (and other information like "valid") and send spam to all of them.

This is going to be a prelude to the command line social networking application we will write in the sequel to this class -- "Bash application development with SQL databases"

Script 1 -- simple while-read

Note that you can memorize the idiom, then figure out what each piece means.

Script 2 -- while-read with complex processing

...

Script 3 -- simple functions

...

Script 4 -- "sourcing" functions with local variables

Stuff


# Take each line of stdin into FOO
#  and do something with it
C=1
while read FOO ; do             
    # files and dirs  that exist in cwd
    if [[ -e $FOO ]]; then
        echo "Got existing: $FOO"
    fi

    case $FOO in
        $PATTERN) echo "got a $PATTERN: $FOO";;
        *) echo "else" > /dev/null;;
    esac
    C=$(( $C + 1 ))
done
echo "Evaluated $C files"

# Nifty thing that could read a database and send out emails
PSQL="/opt/local/lib/postgresql83/bin/psql"
CMD=" select zcta, astext(centroid(the_geom)), 'blah blah' from  zips order by zcta "
$PSQL  postgis_pdx_2008 -F ' ' -A -t -c "$CMD" | while read ZCTA POINT; do
    echo "$ZCTA: $POINT.  Yippee!"
done

Fifth Day

Open discussion -- read manual, questions, future projects, feedback for class, teaching assistants, crazy unix programming (APUE), revision control, look at important FG scripts.

Difference between revisions of "Intro to Bash Scripting"

Revision as of 18:51, 8 November 2008

Contents

Class Description

Class Outline

Class Approach

Code Listings and Class Notes

Every day instructions

First day -- scripts and variables

Script 1

Script 2

Script 3

Script 4

Script 5

Discussion

Second day -- for-loops

Finger exercises

Script 0

Script 1

Script 2

Discussion

Third day -- conditionals

Script 1 -- determine relative order of two numbers, and check input validity

Script 2 -- filter prime numbers

Script 3 -- using case

Discussion

Fourth day -- while-read and functions

Script 1 -- simple while-read

Script 2 -- while-read with complex processing

Script 3 -- simple functions

Script 4 -- "sourcing" functions with local variables

Stuff

Fifth Day

Navigation menu

Search