1

Problem-Solving Approaches in UNIX

When u use a computer, usually have a goal in mind ? To learn more about the system, play a game,write a program ,edit a letter, communicate with email, chat etc An operating system offers you a work environment for satisfying your goals. UNIX has dozens of command that are not standard equipment on most operating systems. Example There are commands that count words, search for strings, sorting etc. UNIX system offers four approaches to problem solving. They range from the very simple to the complex. 1. Simplest approach is to find a UNIX command that directly solves a problem Eg:to sort a file use sort command 2. If u cant find a UNIX command that solves the problem directly look for an appropriate combination of commands. E.g.: If u wish to extract from the file lmcst those lines containing mca and have arrange those lines alphabetically , grep mca lmcst | sort 3. Shell programming (shell scripts) this is a file of unix commands that is used as a program to tell the shell what to do. Also UNIX allows you to use loops etc. 4. sometimes shell script wont solve a problem or else will solve too slowly. The obvious choice on a UNIX system is the C language. Since UNIX is written in C and the UNIX system calls and library functions are available to C programs as ordinary C functions.

4 -problem solving approaches.

Single command solutions combined command solutions shell scripts solutions C program solutions

Using single UNIX command to solve problems Find a command that solves your problem and use it. You may not know about command and how to use it. One solution to these problem is to browse through UNIX manual. The UNIX command line ls -l -s /usr/black/flag

Command name

options

arguments

The simplest command lines consists just of a single command name, such as date or who. You will follow the command name with one or more arguments. These arguments provide additional information for the command. These arguments are filenames or directories.

cd /usr/include another type of argument is the option which modifies the behavior of a command. Sort lmcst // will sort the file contents alphabetically sort -r lmcst // sort in reverse order the options are marked with a hyphen . User can use several options like ls -lsa if you want to extend the command to next line use backslash and hit return.
rajesh@ubuntu:~$ ls -i ff \ > sort 294680 ff 296300 sort The symbol > indicates that the command is being continued to the next line

Standard input, Standard output, Standard error

In a UNIX system, a program, including many UNIX commands, is normally connected automatically to three file called standard input, standard output, standard error. If the program does not explicitly request other files, it gets its input from the standard input, sends output to standard output and sends any error message to the standard error. Unless other wise specified all three files are default to your terminal. (devices are treated as files). ie, it takes input from your key board and sends output to your screen. Unix allows you to change the standard input and output temporarily by using pipes and redirection.

Standard o/p (S.O)

Input comes in here

Unix command
o/p sent out here

Standard input (S.I.)

Error message sent out here

Standard error (S.E)

COMPOUND UNIX COMMANDS

When a single UNIX command does not suffice to solve a problem or do a task, try joining commands together. The chief tools are for this are redirection and the pipe. Redirection Redirection changes the assignments for standard input and standard output. The > operator makes the filename following the operator become the new standard output, and the < operator makes the filename following it the new standard input. cat reads from the standard input (keyboard) if no files is given. $cat Lmcst <controlD> Lmcst We typed cat and hit the <return key>, we provided no input filename. Then what we typed next was gathered as standard input and cat passed it on to the standard output, the terminal screen.
$cat f // not a standard i/p (the internal programming of cat caused the file f to be opened lmcst mcet cet Mit

rajesh@linux-1dzg:~> cat < f lmcst mcet cet mit Both produce the same result. cat like many file-reading programs can read the standard input and a file input during the execution of the command. The standard method is to use a hyphen instead of a filename to stand for standard input. rajesh@linux-1dzg:~> cat Haihai rajesh@linux-1dzg:~> cat f lmcst mcet cet mit unix programming unix programming

cat concatenated the two input. Since the file name came first, it was printed first. Then the keyboard input, represented by the hyphen was printed. $ sort < f > f3
input taken from file f and the out put to be routed to f3 $ cat f3 cet lmcst mcet mit

Another redirection operator is append operator >> cat f f1 f2 f3 >> f4

Pipes Redirection lets us connect programs to files. Pipe facility lets us connect programs to other programs A pipe | makes the output of one program the input of another. grep fresno billsdue | sort the output of grep becomes the input to sort. billsdue
SO

| grep SI Keyboard

SI

SO sort And fresno but fresno SE

SE

Input from pipe can be combined with input from files. Use a hypen symbol , so that the command recognize that there will be a standard input. Grep fresno billsdue | sort olddue > newdue The output from grep becomes the standard input to sort, mean while sort opens the file olddue. The contents of this file are sorted together with the grep output and the sort output is redirected to the file newdue. Redirection routes output to files while pipes route output to other programs.

The TEE
The tee commands reads the standard input and sends it onto the standard output. It also redirects a copy of what it has read into the file of your choice cat part1 part2 | tee total | lpr The o/p of cat becomes the standard input of tee It is sent on through another pipe to the lpr line printer command. Meanwhile a copy is stored in the file total. Append to a file using tee (use -a option) : cat part3 part4 | tee -a total | lpr

Filters
A filter is a program that can take a flow of data form standard input, process (filter) it and send the result on to standard output. Examples

cat, sort, grep How can you tell if a command is a filter? Read its description , if it can take input from the standard input and if it send its output to the standard output, then it is filter.

SHELL SCRIPTS
If user (you) wants to use a sequence of command , he can commit them permanently to a file. Then the shell read the file and follow the script of commands in it, such a file is called a shell script. Example echo Here is the date and time date To run this script, give the filename as argument to the sh (for shell) command. $sh example Here is the date and time Tue Mar 3 19:43:52 PST 2009 Here sh command creates a new child shell process. This new shell reads the file example, executes the command it finds and dies, returning control to the original shell. User can make the example file executable, by giving executable (e) permission to the file example. $Chmod u+x example $example Here is the date and time Tue Mar 3 19:43:52 PST 2009

Features of shell scripts 1. shell scripts can take arguments, just like most commands. These arguments (parameters) can
be symbolically represented within the script. So we can pass on values, filenames. 2. The Shell allows you to create shell variables and to assign values to them. They can be used by shell scripts for many purpose. 3. the shell has a collection of metacharacters (?,*,[]), characters assigned special meanings that greatly expand the applicability of shell scripts.

4. the shell provides control structures such as if .....else constructs and for loops that let you implement programming language approaches. A script built up on standard UNIX commands is highly portable from one UNIX system to another. Shell scripts take up a little storage. The drawback of shell script is that it is relatively slow .

C programs for solving problems

Why C ? Because it is the language that most (90%) of UNIX is written in, so the interface between C and UNIX is straightforward and natural. System calls, the subroutines used by kernel they are C functions. UNIX has about 60 such system functions and they can be used in C programs just like any other C functions. C is a compiled language. Means you use one of the system editors to write a program. Then u submit the program to the C compiler, which converts it to an executable program in machine language. Then u can run the program. Procedures 1. use an editor, such as vi, ex or ed to write the program. File name should end in .C (to identify it is a c program to the compiler example (mca.c) main() { printf (mca @ lmcst \n); } 2. submit the file to cc, the C compiler cc mca.c // if ur program is ok, the compiled version is placed in a file called a.out (by default) 3. to run the program, type a.out $ a.out mca @ lmcst $ 4. to change the name of the a.out file mv a.out mca now just type mca to run the program

Building your own command library of programs

.profile this file is created for you when ur UNIX account is set up. When u login UNIX scans this file for instructions on how to handle your account. (BSD versions use .login) one line in the file is PATH=.: /bin: /usr/bin (PATH is one of the buitl-in shell variable) this provides the list of directories the shell uses to find command names. : is used to separate one directory name from the next. Change the PATH definition PATH=.: /usr/raj/rvs: /bin: /usr/bin once you created this setup you can use your commands the same as you use UNIX commands. Any changes made to .profile do not go into effect until the system reads the file again.

Working with the Bourne shell

shell is the interface between you and the system. Acts as a command interpreter. There are two shells in widespread use the most widespread shell program is the Bourne shell, this is developed by Stephen bourne of bell labs for bell release UNIX. Another shell is csh it is developed by berkely , developed for BSD version of UNIX. Applications developed on Bourne shell can be used for all UNIX systems. Shell's filename expansion capabilities File name expansion is the ability to expand a shorthand notation for a group of filenames to a complete set of explicit filenames. Mr. Mister

Shell metacharacters The shell uses a set of special symbols called metacharacters to let you produce pattern or templates to match various classes of filenames. 1.The ? - stands for any single character. d?g would match three-character filename starting with d and ending with g.

possible matches include dig,dog,d4g, d_8 2. The * - stands for any combination of zero or more characters. (this will not match a period at the beginning of a filename , to s*n match any filenames, regardless of length, that starts with possible matches sn,sun,soon,sudden,s747n,s1.bin
3. The [] - lets u represent a restricted range of single characters

protect .profile) s and ends with n.

enclosed within the brackets. You can use [acdAC] or indicate a range by using a hyphen [a-m]. d[io]g dig , dog (diog is worng) each bracket pair represents just one character place in a name Metacharacters can be used in conjunction with one another. [A-Z]*[0-9] matches any filename starting with a capital letter and ending with a digit. Example : rajesh@ubuntu:~$ ls [A-Z]*[0-9] B2 C8 CZ29 f1 f3 f4 rajesh@ubuntu:~$ ls [A-Z][0-9] B2 C8 f1 f3 f4 *[0-9][0-9] matches any filename ending with two digits. *[10-99] wont work because for a single bracket pair matches only one bracket.

Shell variable
A variable is a name associated with a value. Shell variable comes in two varieties 1. those created and maintained by the UNIX 2. those created by the user

User-Created shell variable name=value (name of shell variable = value of shell variable)

10

pet=bulldog (pet is a shell variable name = its value is the string bulldog shell variables are string variables weight=156 the value of weight is a three-character string consisting of the digit characters 1 5 6. the value of weight is not the number 156. the shell can if necessary interpret a digit-string numerically. One special string called null string, it is a string consists of no characters. Vacuous= ' ' notmuch= idea= No space are used when we assign a value to a variable. You can use letters, digits and underscore character in variable names. Example figaro, HO, name2, go_where (LEGAL VARIABLES) big-foot=squasher (illegal) squasher=big-foot ( legal) To find out the value of a shell variable, we can use echo command $echo pet pet a variable name preceded by a $ as an argument to echo, the value of the variable is echoed. $echo $pet bulldog // pet is a variable name // $pet is the variable's value. $echo the value of pet is $pet the value of pet is bulldog. rajesh@ubuntu:~$ pet= rajesh@ubuntu:~$ echo the name of pet is $pet

11

the name of pet is System shell variables UNIX maintains its own set of shell variables. Type set to see what your system is using. The set command makes the shell print out the variables it knows, (system and user variables). HOME=/usr/mca08 PATH=.:/bin: /usr/bin Left of equal represents the variable name and to the right is the variables value. System shell variables are in upper case. Some standard shell variable EXINIT initialization instructions for the ex and vi editors HOME this is set to the pathname of your home directory IFS (internal field separator) this is set to a list of the characters that are used to separate words in a command line. This list consists of the space character, the tab character and new line character. LOGNAME The users login name MAIL this variables value is the name of the directory in which electronic mail addressed to you is placed. The shell checks the contents of the directory often. PATH this names the directories which the shell will search to find commands that you use. A colon is used to separate the directory names. The directories are searched in the order given. PS1 (Prompt String 1) : this is the symbol used as your prompt ( normally it is set to $, we can redefine it by assigning a new value. PS1=# PS2 (Prompt String 2) : this prompt is used when UNIX thinks you have started a new line with out finishing a command. Example you can continue a line by using a backslash before hitting the return key. rajesh@rajesh-laptop:~$ echo hello ho\ > w are you hello how are you rajesh@rajesh-laptop:~$ TERM this identifies the kind of terminal you use.

12

Many UNIX command use built-in shell variable. cd $HOME we can use system shell variables in programs also Local and Global shell Variables : export ordinarily a shell variable is a local variable. That is , it is known only to the shell that created it. When u start a new shell that shell is born ignorant of the old shell's variables.so each shell variable are private (local) to itself. rajesh@rajesh-laptop:~$ cat=kate rajesh@rajesh-laptop:~$ echo $cat kate rajesh@rajesh-laptop:~$ sh $ echo $cat $ rajesh@rajesh-laptop:~$ echo $cat kate If we want the new shell to know the old's shells variable use the export command. Any shell variable used as an argument for this command will have copies of the variable and its value presented to all shells descending from it.this kind of variable is termed global. rajesh@rajesh-laptop:~$ export cat rajesh@rajesh-laptop:~$ cat=lmcst rajesh@rajesh-laptop:~$ echo $cat lmcst rajesh@rajesh-laptop:~$ sh $ echo $cat lmcst $ rajesh@rajesh-laptop:~$ echo $cat lmcst rajesh@rajesh-laptop:~$ sh $ export j $ j=john $ echo $j john $ rajesh@rajesh-laptop:~$ echo $j Variables can be exported to subshells, but they cannot be exported back to parent shells. Exported shells are not completely global variables

SHELL SCRIPTS
two types of approaches

13

1. write an interactive script that requests the information it needs from the user. Prompts the user fro input, perhaps asking a question, and then reads the users answer. (shell is an interactive program it prompts with $ and reads the commands you type) 2. the script accepts arguments from from the command line. cat sortfile this method is more powerful and flexible. Interactive Shell Scripts this is based on using UNIX shell commands read and echo. echo Dear User , what is ur name\? read name echo ur name is $name here we use the backslash to remove the special pattern-matching property of the question mark character. (this is called escaping or quoting the character) place this program in a file glad and execute. Echo enter three nos read a b c the first variable gets the first value u type and so on if there are more words than variable,all the left over words get assigned to the last variable. echo enter 3 nos read a b c echo $a echo $b echo $c echo $a - $c enter 3 nos 123456 1 2 3456 1-3456

If there are more variables than words, the unused variables are not assigned values. Read is a built in shell command., it is part of the sh program program for copying a file

14

echo what file to copy read org echo what name u desire for copy read copy cp $org $copy echo $org is copied into $copy

Shell scripts with arguments

positional parameters convert (shell script - $0) pounds $1 yen $2 pesos $3

refer the first argument as $1 , the second argument as $2, the special symbol $0 stands for the name of the command itself. $* stands for all the arguments from $1 on up ($1,$2,$3 ) the numbered arguments are referred to as positional parameters $#, which represents the number of arguments . rajesh@rajesh-laptop:~/Desktop$ sh show a b c I used show command thses are my arguments a b c the total arguments are 3 the first arg is a the second arg is b the third arg is c cp $1 $HOME/Music/$1.sv rajesh@rajesh-laptop:~/Desktop$ sh new\ file hh rajesh@rajesh-laptop:~/Desktop$ ls /home/rajesh/Music/ hh.sv we can use $1 more than once. Write a shell script to make a shell script into an execuitable. chmod u+x $1 // save this in a file execfile

rajesh@rajesh-laptop:~/Desktop$ sh execfile hh copy this execfile in /usr/bin then u just type execfile hh using set set is used to assign values to positional parameters. rajesh@rajesh-laptop:~/Desktop$ set hai how r u rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3 $4

15

hai how r u Set is used in conjunction with the metacharacter ` (backquote) when a command is enclosed in backquotes, the command is replaced by the output it produces, this process is called command substitution. rajesh@rajesh-laptop:~/Desktop$ echo date date rajesh@rajesh-laptop:~/Desktop$ echo `date` Thu Mar 12 02:51:04 IST 2009 rajesh@rajesh-laptop:~/Desktop$ who am i rajesh pts/0 2009-03-12 01:43 (:0.0) rajesh@rajesh-laptop:~/Desktop$ set who am i rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3 who am i rajesh@rajesh-laptop:~/Desktop$ set `who am i` rajesh@rajesh-laptop:~/Desktop$ echo $1 $2 $3 $4 $5 rajesh pts/0 2009-03-12 01:43 (:0.0) Write a shell script that print outs date information in this order : time , day of week, day number, month, year. rajesh@rajesh-laptop:~/Desktop$ set `date` rajesh@rajesh-laptop:~/Desktop$ echo `date` Thu Mar 12 03:28:25 IST 2009 rajesh@rajesh-laptop:~/Desktop$ echo $4 $1 $3 $2 $6 03:28:32 Thu 12 Mar 2009 Eg)echo do not panic! All is well name=$1 echo you know $name, that are one of my favourite users. set `date` echo It is $1 $3 th $4 , $6 echo relax $name echo the time is now $4 $5 do not panic! All is well you know rajesh, that are one of my favourite users. It is Thu 12 th 03:38:07 , 2009 relax rajesh the time is now 03:38:07 IST

16

eg)set `who` who | grep -i $1 echo $1 if [ $1 = rajesh ] then echo the user $1 logged in fi rajesh@rajesh-laptop:~/Desktop$ sh user rajesh tty7 2009-03-12 03:59 (:0) rajesh pts/0 2009-03-12 04:03 (:0.0) rajesh the user rajesh logged in Looping and making choice what truly make the shell a programming language is its ability to control the flow of a program for loop case while until if constructions THE for LOOP a loop is a programming device that lets you cycle through the same steps several times. The general form of the Bourne shell for loop for variable-name in value1 value2 do commands done --------------------------------------------------------------------------------------------------Eg)for i in 1 2 3 4 do echo $i done 1 2 3 4

17

Eg)for i in 1 2 3 4; do echo $i; done Eg)for i in 1 2 3 4 > do > echo $i > done Eg)for i in fly spider frog > do > echo i know $i > echo swallowed a $i > done i know fly swallowed a fly i know spider swallowed a spider i know frog swallowed a frog Printing file contents one word per line for i in `cat glad` do echo $i done For loop with command line argument for i in $* do echo $i done rajesh@rajesh-laptop:~$ sh hh 12 34 5 6 12 34 5 6 Eg)for i do echo $i done eg)for i in $@ do echo $i

18

done Generating Values for a for Loop 1. for i in 2. for i in $* for i 3. for file in * do cat $file done 4. take from shell variable // interactive program echo Enter ur name read name for i in $name do echo $i done 5. take value from the output of a command rajesh@rajesh-laptop:~$ for i in `cat jj` do echo $i done output echo Enter ur name read name for echo $i done Some times for loop is invaluable when you want to process several files in some manner. To do a word count on several files wc ex hh jj for loop is invaluable at this time (wc command has a loop mechanism built in). -m, --chars

19

print the character counts -l, --lines print the newline counts -w, --words print the word counts To sort multiplte files for i in $* do sort $i done

Word-seeking program

#wordseek-searches files for words (to add comments) source=$1 shift //drops the original $1 from the argument list, leaving $* equal to the rest of the arguments for i in `cat $source` //getting words from source file do grep $i $* // Si is the word , S* are the filenames done sh wordseek music ch[1-3] CHOICE-MAKING : THE case STATEMENT Syntax case value in choice1) choice2) .... esac commands; commands;

Choice1, choice2 are labels that identify potential choices of action. One use for the case statement is to set up menus Example echo " BASIC UNIX COMMANDS " echo "--------------------------" echo "command option" echo "__________________________"

20

echo "LIST 1" echo "DATE 2" echo "CURRENT DIRECTORY echo "EXIT 4" echo "--------------------------" echo "Enter your option.." read option case $option in 1)ls ;; 2) who am i ;; p|3) pwd ;; quit|4) exit ;; *) echo invalid choice ;; esac

3"

We can use command-line argument for case statement echo " BASIC UNIX COMMANDS " echo "--------------------------" echo "command option" echo "__________________________" echo "LIST 1" echo "DATE 2" echo "CURRENT DIRECTORY 3" echo "EXIT 4" echo "--------------------------" echo "Enter your option.." read option case $1 in 1)ls ;; 2) who am i ;; p|3) pwd ;; quit|4) exit ;; *) echo invalid choice ;; esac Pattern Matching we can use ? And [] metacharacters. case $1 in [aeiouAEIOU]*) echo the given word starting with vowel ;; ??) echo a two letter word ;; *) echo I dont know ;; esac $ sh vowels ears the given word starting with vowel

21

$ sh vowels EYE the given word starting with vowel $ sh vowels ZZ a two letter word $ sh vowels 1323 I dont know Make the file execuitable and run $vowels eye

We can use case `pwd` in using for and case together for file in * do case $file in *.c) echo it is a c file $file;; *.cc) echo it is a c++ file $file;; *.txt) echo it is a text file $file;; esac done # sh forcase it is a c file a.c it is a c++ file a.cc it is a text file a.txt it is a c file b.c it is a text file new file.txt

CONDITIONAL LOOPING : while AND until

UNIX offers three loop structures : for, while and until.The general form of while loop is while control command do commands done echo enter the choice read i while [ $i -gt 10 ] do echo $i no greater than 10 done //while loop like C or PASCAL is wrong cannt use < > == (conditions)

22

If the control command returns a zero exit status , the command between do and done are executed. Word=$1 //whilegrep shift while grep $word $1 do shift done echo first file not containing $word First variable word gets the value arrahman. The shift command shfits ch1 to $1 the while statement become while grep arrahman ch1 do shift done If the grep command reports success, then the instructions between do and done are executed., if it reports failure the loop is terminated. Suppose arrahman is found in ch1, then the grep reports success, and the following shift command is executed , this shifts ch2 to $1 and the next loop becomes while grep arrahman ch2 do shift done if the grep fails there is no arrahman in ch2 the loop terminates the script move onto next line echo first file not containing $word The Until loop The until loop is executed until the test condition succeeds. A while loop runs until a control command fails The until loop runs until a control command succeeds until control command do commands done until ls | grep $1 > /dev/null do sleep 30 //loop continues until the file is created.

23

done echo file $1 is here THE if STATEMENT if statement is concerned with the status of commands if control command then commands fi first the command following the if is executed. If the command is successful (exit status 0), then the commands between the then and the fi are performed. if cmp -s $1 $2 then rm $2 fi The if .....then.....else structure if control command then commands else commands fi eg)if cmp -s $1 $2 then rm $2 echo the $2 was duplicate and removed else echo the files $1 and $2 are not the same fi If then .....elif then else if control command 1 then commands elif control command2 then commands else commands fi // -s suppresses the usual output only the status of the command (0 if the file match, 1 if they dont (fail)

24

eg)if [ $leap -eq 1 -a $date -gt 29 ] then echo "date not valid" elif [ $leap -eq 0 -a $date -gt 28 ] then echo "date not valid" echo "enter correct date and try" else echo "Date valid" fi

THE test COMMAND

$ dog=nikkit //set dog variable to nikki test $dog=nikki // test for equality the test command produces no o/p, but it produces a 0 exit status if the test is successful. And a 1 exit status if it fails. $? check status of test command Eg)$dog=nikki $test $dog = nikki $echo $? 0 $test $dog = pluto $echo $? 1 Test string1 = string2 (There is a space between two strings)
Eg)Echo what is the best operating system if test $1 = UNIX then echo I agree else echo I never heard of $1 fi eg) Echo i am thinking of a number between 1 and 50 read guess until test $guess = 33 do echo nope guess again read guess done echo well done // test

25 File checking the test command has several options for checking the status of a file -r file - true if the file exists and readable -w file - true if the file exists and writable -f file - true if the file exists and is not a directory -d file - true if the file exists and is a directory -s file - true if the file exists and has a size greater than zero -x for identifying executable files $ ls -l ex -rw-r--r-- 1 rajesh rajesh 8 2009-03-15 20:17 ex $ test -r ex $ echo $? 0 $ test -x ex $ echo $? 1

String Checking comparing two strings string1 = string2 // true if the strings are same string1 != string2 // true if the strings differ the space are obligatory test $ans = hoop //valid form test $ans= hoop //invalid form three options deal with the size of a string -n string // true if the string has nonzero lenght -z string // true if the string has zero length string // true if the string is not the null string An undefined variable , appears to be a null string $ frog=bumpy $ test $frog $ echo $? 0 $ test $frag //frag not defined $ echo $? 1 We can use test to make a while loop act like a foor loop while test $1 do .....

26

shift done Numerical Comparisons thses options compare integers n1 -eq n2 // true if the integers are equal n1 -ne n2 // true if the integers are not equal n1 -gt n2 // true if n1 is greater than n2 n1 -lt n2 // true if n1 is less than n2 n1 -ge n2 // true if n1 is greater than or equal to n2 n1 -le n2 // true if n1 is less than or equal to n2 eg)if test $# -ne 2 then echo this pgm needs two args $ sh numeric 1 this pgm needs two args $ sh numeric 1 2 5 = 5 // true , the character string 5 is same as 5 5 = 05 // false , the character string '5' is different from '05' 5 -eq 05 // true , 5 and 05 have the same numerical value $ test 5 = 5 $ echo $? 0 $ test 5 = 05 $ echo $? 1 $ test 5 -eq 05 $ echo $? 0 Logical operations test -rw flees is it possible ? ! -a binary and operator -o binary or operator test ! -r film // true if the filem is not an existing, readable file test -r film -a -w film // true if the file film is readable and writable

27

test -r film -o -w film // true if the file film is readable or writable Using parentheses for grouping test \( -r film -a -w film \) -o \( -r flam -a -w flam \) parentheses are shell metacharacters so use the backslash Test with [ ] echo what is ur age read age if [ $age -le 6 ] then echo child --------------------------------------------------if [ $age -gt 9 -a $age -lt 20 ] // write a shell script to check whether the given file is exists or not if test ! -f $1 then echo file $1 not found else echo file found fi

UNIX Tools : grep, sed, tr, and awk

grep a pattern matching utility sed a stream editor tr a character translation program awk a pattern matching and processing labguage Grep grep stands for global regular expression printer grep pattern filename example grep mca lmcst grep seraches the string mca in the file lmcst and print the line containing the string $ ls | grep a Examples glad ha Templates Common options for grep -v prints those lines that do not match -c prints only a count of the number of matching lines -l prints only the names of files with matching lines -n prints the line number with each output line -s prints no o/p, just return status

28

$ grep -v mca lmcst mc $ grep -c mca lmcst 3 $ grep -l mca lmcst lmcst $ cat lmcst mca is a pg course mca master of computer applications - mca mc Grep accepts multiple filename grep mca lmcst lmcst1 lmcst2 If the pattern contains a space or tab character in it, it is necessary to put them on in a single or double quotes. grep 'san francisco' hotels or grep san francisco hotels Greps pattern-matching $ grep ar c? c1:haiarrahman c2:arrahman oscar grep 'm.a' lmcst // will count the space also a period represents any one character eg)rajesh@rajesh-laptop:~$ cat > grep spate spot respite sputter the asp took rajesh@rajesh-laptop:~$ grep 'sp.t' grep spate spot respite sputter the asp took rajesh@rajesh-laptop:~$ grep 'sp..t' grep

29

sputter Examples h.t // matches hit, hst, h8t h[ea]t // het , hat (heat is not matched) [A-Z][^ A-Z] // two character first is a capital letter second is not a capital letter (examples B4,Me,S!) ^The // matches any line beginning with The , caret symbol is different inside and outside the bracket \.$ // matches any line ending with period ho*t // matches ht(zero occurrences of the preceding pattern),hot,hoot,hooot and so on

Some typical pattern-matching problem

1. counting blank lines to count the number of blank lines in a file, first we use greps -c option, which counts the number of matche and Second we need a pattern to match a blank line ^$ // matches an empty line $ grep -c ^$ lmcst 2 when we write a file , we start the first line by hitting the space bar. In this case there will be some space character present but invisible to find the spaces we can use *$ a blank line can also contain a tab character , to look for lines containing just tabs, we can use this pattern: *$ //matches blank line with any number of tabs In C programs we can find the tab character by using the pattern ^\t*$ we can combine these patterns to catch lines that may have blank lines,tabs or both ^[ \t] * $ to count the blank lines (with or without spaces or tabs) in the file text is grep -c '^[ \t]*$' text 2. finding two patterns in proper sequence to find all lines containing Mickey and Donald in that order, but which may have additional character in between them. then Mickey tossed the gem to Donald grep 'Mickey.*Donald' hero rajesh@rajesh-laptop:~$ cat > sp the Mickey tossed the gem to Donald Mickey and Donald are friends rajesh@rajesh-laptop:~$ grep 'Mickey.*Donald' sp the Mickey tossed the gem to Donald

30

Mickey and Donald are friends the pattern .* stands for zero or more occurrences of any character We didnt care about which order the two words came in, we could do this grep Mickey hero | grep Donald the first pass through grep produces only line containing Mickey, the pipe makes the list of lines containing Mickey into input for the second pass through grep, and the only lines making it through both passes are the ones containing both Mickey and Donald. 3. finding a word grep find strings, to serach for a particular word grep has a -w option fgrep and egrep fgrep command doesnt use any special character (like * or $) fgrep is faster fgrep can search for several strings simultaneously egrep can also search for several strings simultaneously both command offers a -f option that lets them take the search pattern from a file instead of command line. rajesh@rajesh-laptop:~$ fgrep 'mca > mb > mba', lmcst mca is a pg course mca mba mbamca mb eg)rajesh@rajesh-laptop:~$ egrep 'mca|mba|mb' lmcst mca is a pg course mca master of computer applications - mca mcmba mba mbamca mb Marliyn|James Monore this means Marliyn or James Monore to change the grouping parenthesis can be used (Marilyn|James) Monore matches both Marilyn Monore and James Monore

31

Two other additions to egrep are the special + and ?, both are similiar to * , which stood for zero or more occurrences of the preceding character. The + stands for 1 or more occurrences and the ? Stands for zero or one occurrence. ho*t matches ht,hot,hoot etc ho+t matches hot,hoot etc ho?t matches ht and hot parentheses can be used with *, + , ? (ho)*t matches t,hot,hohot etc pattern using the egrep scheme of special characters are known as full regualr expressions. CHARACTER TRANSLITERATION : tr tr transliterates characters $cat philo I compute, therefore I am. $ tr eiou ~#$% < philo I c$mp%t~, th~r~f$r~ I am. tr is a pure filter, it takes input from the standard input. Small letters (i) and capital letters (I) are different, above example I is not affected by I $ tr eiou EU < philo I cUmpUtE, thErEfUrE I am Special Notations you can indicate ranges with tr by using a hyphen tr A-Z a-z //converts upper case to lower case or tr [a-z].[A-Z] Options tr command has three options -d uses just one list of characters, and it causes them to delete $tr -d eiou < philo I cmpt, thrfr I am -c option causes the substitutions to affect every character but those in the first list. $ tr eiou * < philo I c*mp*t*, th*r*f*r* I am $ tr -c eiou '*' < philo ***o**u*e****e*e*o*e******$ -c option replaces all the space,punctuation with * and new line character at the end

32

finally -s option compresses strings of repeated ouput characters at the end of the line, ie strings **** become *. $ tr -cs eiou '*' < philo *o*u*e*e*e*o*e*$ rajesh@rajesh-laptop:~$ cat > philo I compute , therefore rajesh@rajesh-laptop:~$ cat philo | tr "[a-z]" "[A-Z]" > ph1 rajesh@rajesh-laptop:~$ cat ph1 I COMPUTE , THEREFORE $ tr eiou EU < philo I cUmpUtE, thErEfUrE I am the last character from the second list is used for all the left over characters from the first list. THE STREAM EDITOR : sed to edit a file called philo u type ed philo the editor then creates a temporary copy of philo in the /tmp directory, all the changes made to the copy, when we give the w command, the temporary file is copied into the original, the net result is that the editor alters the original file. Sed is a UNIX filter, it can take input from a file, but its output is sent to the standard output, the original file is not altered. The o/p from sed can be redirected to a new file if u wish to save it. Or it can be piped along to some other program as input. What exactly does sed do? It takes a line from input, applies an editing command to it, and prints the result. Then it takes the next line and repeats the process until the whole file has flowed through sed. Basic usage of sed is sed 'editing command' filename if no filename is given, then sed reads the standard input, this means it can receive input from a pipe. If more than one filename is given, then they are processed in turn. Editing commands for sed it has two parts : a line specifier and an editing instruction if there are no line specifier then the command is applied to every line. There are two approaches to specifying lines. One way to specify a line is to give a line number or a range of line numbers. 3 // third line 5,8 // lines 5 through 8 10, $ // lines 10 through the last line

33

second way to specify a line for sed is to provide a matching pattern enclosed in slashes. (patterns are the same as those used by grep) Examples /huge/ all lines containing the string huge /[Dd]o/ all lines containing strings Do or do /^[ \t]*$/ all blank lines. The two forms of line identification (line numbering and pattern matching) can be combined. 1,/gossip/ From line 1 to the first mention of gossip ! - to indicate the command is to be applied to lines that dont match the line specifier. The editing instructions for sed s substitute p print d delete q quit editing instructions immediately follows the line specifier sed '10q' text // quit after reading line 10 sed prints each line as it is processed, this command prints the first 10 lines of a file. Sed '$p' text //prints the last line sed '/^[ \t]*$/d' text // delete all blank lines from text Sed '/value/!d' text // delete every line in text not containing the string value The sed substitution command (s) the general form is s/oldstring/newstring this replaces the first occurrence of oldstring on a line with newstring. If u want to replace all occurrences on a given line we tack on a g (for global) at the end of the command. Sed 's/Michael/Michele/g' text $ cat > text john is coming from tvm john and his friend john $ sed 's/john/hary/' text hary is coming from tvm hary and his friend john $ sed 's/john/hary/g' text hary is coming from tvm hary and his friend hary Sed '1, /Jessup/s/Michael/Michele/g' text first line containing Jessup // replaces Michael with Michele up through the

34

special characters also can be used with sed $ sed 's/john/"hary"/g' text "hary" is coming from tvm "hary" and his friend "hary" $ sed 's/john/"john"/g' text "john" is coming from tvm "john" and his friend "john" $ sed 's/john/"&"/g' text "john" is coming from tvm "john" and his friend "john" $ sed 's/john/&! &!/g' text john! john! is coming from tvm john! john! and his friend john! john! Sed tags you can enclose part of oldstring between \( and \) the enclosed part is said to be tagged. U can then refer in newstring to the tagged part as \1 if u tag more than one part in old string, u can call the second tagged part \2, and so on s/ \ (john\) and \(hary\) / \2 and \1/ - replaces the phrase john and hary with hary and john. \1 stands for john (first tagged string) \2 stands for hary (second tagged string) $ sed 's/ \(john\) is \(coming\) / \2 is \1/g' text coming is johnfrom tvm john and his friend john To change all occurrences of occurence and Occurence to occurence and Occurence s/ [Oo]ccurence/ [Oo]ccurence/g this dont work the pattern [Oo] can only be used in the left-hand pattern (oldstring) the only special symbols usable on the right side (new string) are & and \1 family. So patterns using [], . , * cant be used on the right side. S/ \ ([Oo]ccur \) ence / \ 1rence / g $ sed 's/john/hary/g > s/hary/mary/g > s/mary/bony/g' text bony is coming from tvm bony and his friend bony ------------------------------------------------------$ sed 's/john/hary/g s/is/was/g' text

35

hary was coming from tvm hary and hwas friend hary Options for sed -n , -f , -e -n option suppresses the automatic printing of each processed line, Sed -n '$p' list it print just the last line of the file list the -f option followed by a space, then a filename, This option causes sed to take its editing commands from the named file sed -f edit sermon sed to look in the file edit for commands to apply to the file sermon. Sed -f name -f form memo this command first use the editing command from name and then from memo the -e option the -e option lets you mix a command-line editing command with commands in a file. Sed -e 's/Boston/Wasco/g' -f form trav.txt Boston will replaced by Wasco then the editing command in form to be applied, and then on to the next line.

A pattern-scanning and processing language : awk

The awk command (named after and by its creators : Aho, Weinberger, and Kernighan) awk command combines pattern matching, comparison making, numerical operations and c-like programming features into one program. A basic command line looks like awk awk-command filename each awk command has a line-specifier (a pattern) part followed by an instruction (an action). Awk lets you identify particular fields within a line a field is a squence of characters containing no blanks & separated by blanks from other fields. Each word on this line is a separate field. Example of an awk command line. awk '/poultry/ { print $1, $3, $6}' supplies the pattern part is /poultry/. The action part is enclosed in braces; it requests that the first, third and sixth fields of the matching line be printed. The $1 stands for first field, $3 stands for third field if there is no pattern portion the action is applied to every line of input. The following prints the first & second fields of the table supplies in reversed order. awk ' { print $2, $1}' supplies

36

If there is no action portion, the default action is to print the entire line which matches the pattern awk '/poultry/ ' supplies more than one command can be included by starting a new line for each command $ awk '/scandel/ { print $0} /rumour/ { print $0} ' rag $0 stands for entire line Eg)$ awk '/mca/ {print $0} > /john/ {print $0} ' lmcst mca hai hai mca hai john mca hai john mca eg)$ awk '/mca/ {print $1} /john/ {print $0} ' lmcst mca mca hai hai john mca If u have many commands, it is usually more convenient to place them in a file and the awk get its commands from the file. This is done by using the -f option. awk -f cawk rag this command tells awk to apply the command in the cawk file to the input from the rag file. Fields and Records an awk program processes its input a record at a time. By default a record is just a line. The new line character is the default record separator. When ever awk encounters a newline character it starts a new record. Within a record, the input is divided into fields. A field is a series of characters delineated by a fields separator, by default the field separator is a blank, that is a space or tab. Example mc |mcst Btech CSE lmcst lmcst is the second field in the first record Btech is the first field in the second record. Awk has some built-in variables dealing with the fields and records. NR represents the current record no. NF represents the number of fields in the current record. In the above example after reading first line NR =1 and NF = 2

37

the variable FS stands for field separator, by default this is blank (ie a space or tab character) we can reset this field separator to another value. This can be done by using -F awk -F: '{print $1, $2}' lmcst this prints the first two fields of the lmcst file, using a colon as a field separator for the input. The variable RS represents the record separator Pattern-Matching Metacharacters for awk \ Match following character literally ^ the beginning of a line $ the end of a line . any one character [list] any one character in list [^list] any one character not in list * zero or more occurrences of the preceding character or group + one or more occurrences of the preceding character or group ? zero or more occurrences of the preceding character or group (exp) is exp exp1 \ exp2 means exp1 or exp2 Patterns in awk the awk command accepts several forms of pattern-matching Full Regular Expressions awk recognizes the same full regular expressions that grep does. Regular Expressions and Tables the entire line is called $0. the ~ symbols indicate a match the !~ symbols indicate no match to find all lines in which the second field matches flubbo $2 ~ /Flubbo/ $2 ~ /^flubbo$/ this matches only lines in which the second field is flubbo. $2 !~ /^Noxin$/ matches all lines for which the second field is not Noxin Relational Expressions this compare two expressions awk command recognizes the same relational operators that C does. $ cat > compare 12 34 44 12 55 23 $ awk '$2 > $3' compare

38

12 55 23 // prints the line $2 greater than $3 $ awk '$2 < $3' compare 12 34 44 // // prints the line $2 less than $3 awk also recognizes the arithmetic operators +, - , * , / and %. $ awk '$2 >=2 *$3' compare 12 55 23 // print line for which $2 greater than or equal to two times of $3 Relational operators for awk Operator Meaning < less than <= less than or equal to > greater than >= greater than or equal to == equal to != not equal to Logical operators for awk && and || or ! not $ awk '$2 > $1 && $3 > $1' compare 12 33 44 11 23 45 !($2 > $1 && $3 > $1) // selects records for which the condition is not true. Ranges awk allows you to specify a range of lines. NR == 20, NR == 40 // the records 20 through 40 NR == 1, /[D]dismay/ // record 1 through the first appearance of Dismay or dismay Special Patterns awk has two special patterns BEGIN and END BEGIN is used to label actions to be done before any records are read. BEGIN {RS = :; FS = /} this command establishes the colon as the record separator and sets up slash as the field separator. The END label is used to identify actions that are done after all the input has been read. Actions the awk actions can print, it also has built in functions, ability to create variables , arrays, arithmetic operations and flow control statements. Printing to use print statement $ awk '{print $1}' lmcst mca

39

mca hai hai eg)$ awk '{print $1, $2}' lmcst mca hai mca hai hai john eg)$ awk '{print $1 + $2}' compare 45 34 Eg)$ awk '{print}' compare 12 33 44 11 23 45 Eg)$ awk '{print $0, "total is: " , $1 + $2 + $3}' compare 12 33 44 total is: 89 11 23 45 total is: 79 Eg)$ awk '{printf "first val is: %6d second val is %d: total is: $%d\n", $1, $2, $1 + $2}' compare first val is: 12 second val is 33: total is: $45 first val is: 11 second val is 23: total is: $34 Print Redirection awk can redirect print output to files $ awk '{print $1 >> "col1"; print $2 >> "col2" }' compare the first field in each line is appended to a file named col1 the second field will go to the file named col2 Variables awk lets us create a variable, the value of a variable can be numeric or a string. We want to count the number of lines that begin with mca or with hai $ awk '$1 == "mca" || $1 == "hai" { count = count + 1 } END { print count, "lines"}' lmcst 4 lines We can use { count +=1} and {count++} += operator adds whatever is to its right to whatever is to its left ++ operator increments its variable by 1 you have a table of numbers and wish to total the third column $ cat compare 12 33 44 11 23 45 $ awk '{sum += $3}

40

> END {print sum}' compare 89 Arrays $ awk '{sum[1] += $1; sum[2] += $2; sum[3] = sum[1] + sum[2]} END {print sum[1], sum[2], sum[3]}' compare 23 56 79 this creates an array sum with three elements sim[1], sum[2], sum[3] the array index can be a non-null value (including a string) we can use tom and dick instead of 1 and 2 to find how often the words philodox and snoach occur in a file /philodox/ { count[philodox]++} /snoach/ { count[snoach]++} END { print count[philidox], count[snoach]} Flow Control awk uses the if , if-else, for and while loops of C The if statement { if ( $1 < $2) print $1, $2 else print $2, $1 } $ awk '{ if ($1 < $2) print $1, $2 else print $2, $1 }' compare 12 34 11 55 23 Eg)$ awk '{ if ($1 < $2) print $1, $2 }' compare 12 34 23 The for statement the for statement has this form : for (initialize; test; update) command the initialize expression is performed once, when the loop starts up. Then the test expression is examined, if it is true, the command is executed, if it is false the loop terminated. Each time loop is executed the update statement gets executed.

41

A non-C form of for loop for ( i in array) command array is an array name , the command is performed for each array element. { count[$1]++} END { for (i in count) print i, length(i), count[i] } $ awk -f awkex lmcst hai 3 2 mca 3 2 lmcst 5 1 lm 2 1 awkex $ cat lmcst mca hai hai mca hai hai john mca lm lmcst

The while loop the while loop has the form: while ( test ) command

// the command get executed as long as the test is true.

Prgm this prints out the fields in a line until it hits the end of the line or the word private. cat > sample johns hobby is swimming ben's hobbies are fishing private sewing toe-picking jillians hobbies are volleyball { field = 1 while ( field <= NF && $field != "private" ) { printf "%s ", $field field++ } printf "\n" } NF number of fileds

awkex

42

awk -f awkex sample johns hobby is swimming ben's hobbies are fishing jillians hobbies are volleyball the program uses the variable field to keep track of the field number. The while loop ends when it reaches the last field number (field == NF) or when the value of the field ($field) is private. Jumps : continue, break, next and exit break causes the loop to break off and terminate continue causes loop to skip to the beginning of the next iteration next causes awk to skip to the next record, restarting the action there. exit takes you to the end of the input, but does not skip over an END statement. Sprintf sprintf function lets you format various expressions into a string. It uses the formatting conventions of C's printf() Ex: $1 has the value Dan and $3 has the value 65 then label = sprintf(%s of the %d kg class, $1,$3) Results in the variable label having the value Dan of the 65 kg class