Shell: benefits

general-purpose application. An app for running apps.
1. generate filenames from a user-specified pattern (wildcards).
2. input and output can be redirected to/from files and programs, not just terminal.
3. create your own customized environment (shell options, aliases, variables, scripts, functions)
4. programming on the command line
Shell is a program that prompts for a command line, interprets it, runs program.

shell metacharacters (operators of the shell language)

Filename generation: (globbing) match existing filenames

  *  match any string of chars, including null string.
  ?  match any single char.        * and ? are called wildcards.
 []  match any one of the chars enclosed.  A pair of chars,
  separated by a minus will match any char in ASCII order. [abc] [a-z]
  [!]  or [^] match any char except one of the enclosed. [^a-z]

Generated filenames are in ASCII order.

Example patterns:

*     all files
f*    all files in current directory that start with f, including f
*f    all files that end with f, incl. f
f?*   all files in current directory that start with f and have at
         least one other char
[a-z]*        start with a thru z
[a-z]         single lowercase letter filenames
f[aeiou]?  start with f, followed by a vowel, followed by one char
*.gif          all files ending in .gif
*.???         all files ending in a DOS-like extension
/etc/rc5.d/S*  files in that directory that start with S

Ex. commands

$ ls -l *.cpp           #files that end in .cpp
$ grep rand *.cpp   #search for rand in all C++ programs
$ ls -l [a-z]*[0-9] #lists files that start with lower letter, end with digit, anything, or nothing, between.
$ rm *[0-9][0-9]    #deletes all files whose names end in two digits

$ rm *[10-99]       Warning: shell doesn't know numbers.  This is not ten thru ninety-nine.  1 or 0 thru 9 or 9 (poorly written, is equivalent to [0-9]).   ASCII sequence only.
[1-9][0-9]

$ rm * [0-9][0-9]       
Warning: space after *, will match all files and delete them all!

Exception: * won't list files that start with . (dotfiles, so-called hidden files). Must do .* to generate their names. Such files are not usually listed by ls (use -a option of ls); mostly are configuration files (customizable startup) for various apps and shell.

Every directory has these two filenames (links): [to access their inodes]
. this directory. [so can open cwd without knowing its pathname]
.. its parent directory

Exs. file1 == ./file1

$ cp /home/billybob/file1 .       #copy into current directory
$ cd ..          # move up a directory
$ cp ../file .    #copy file in parent directory to this directory
$ mv ../*.h .  #move the .h files from parent directory to this one
$ mv *.h ..        #move them back up

echo command displays its arguments. Useful for seeing (and learning) shell's interpretation of command line. also to display value of shell variables and as print statement of scripts.

$ echo hello world
hello world
$ echo hello           world      #2 arguments
hello world
$ echo f*                # shell expands to list of all matching files
f1 fil2 fred.c         # all the filenames that start with f 
$ echo *                # all filenames
f1 fil2 fred.c mystuff junk 
$ echo *[0-9][0-9]
$ echo asdfasdf >afile

If no filenames match the pattern, the shell passes the pattern itself:

$ echo f*5
f*5
$ echo *     #in an empty directory
*

Command line is given to shell, shell sees * or other filename matching chars, expands to the list of files, then gives the list to the command as arguments. The command never sees the pattern matching chars! The pattern matching chars are not arguments to the command.

User command line   After shell processing
    ls f*       ls f1 fil2 fred.c
    f*      f1 fil2 fred.c   
                      (probably error, would try to execute command f1)

-----------------------------------------------------------------------------------
Shell order of evaluation of command line: simplified

keywords (if, case, for, function, while, !, {} ...)
aliases
built-in commands (alias, cd, echo, export, history, jobs, kill, read, set, shift, test, trap, umask ...)
functions
PATH search for command

-----------------------------------------------------------------------------------
Shell options. set -o list on/off values

noclobber can't redirect to existing file
ignoreeof can't ^D logout
emacs using emacs editing mode
trackall

**************************************************
Input-output redirection.
Most commands take input from standard input, and send output to standard output. Defaults are keyboard and monitor, i.e. the terminal [device file of terminal]. But can change what the standard input and standard output are.
I/O "instructions":
> redirect output to file (Warning: wipes out current contents of file, if any, unless noclobber is on).
< redirect input to come from file
>> redirect output (append) to file
2> redirect standard error to file

$ cat f1 >f2         # copy f1 to f2
$ cat f3 >>f2     # append f3 onto f2
$ cat <f1            # display f1
$ cat f1 f2 f3 >f4   # all to f4
$ cat f1 f2 f3 >>f4   # all appended to f4
$ cat f1 >dir1       # ERROR.
$ mail billybob <letter3 # letter3 is the message
$ cat f1 >f1         # wipes out f1
$ cat <f1 >f1     # Warning.  creation of f1 wipes out contents
$ cat f1 f2 >f2      # net result is copy of f1 in f2
$ cat f1 >>f1 # not allowed: won't append f1 to itself  
                [if f1 not exist creates empty]
$ cat <f1 >>f1     # not allowed.  recurses?!
$cat f* >f5
$ sort xfile >xfilesorted   
$ grep math *.cpp >mf   #all lines containing 'math' from all the 
                    C++ files into file mf
$ file * >filetypes     #type of all files

N.B.
 cat f1     filename argument
 cat <f1 redirect input so comes from file f1, cat command does 
        not see redirection. No file argument for cat command here.
 cat >f2     output redirected to file f2.  No file arg to cat.
  What's the input? Keyboard or mouse paste.  Will accept input until ^D.

If no filenames specified to command, will take input from standard input (allowing input redirection and piping).

Shell does the redirection, command program is oblivious as to where standard input is coming from (terminal or redirected file) and to where standard output is going to (terminal or file).

Shell doesn't care about format. Symbols and filenames removed from command line.

$ >f2 cat<f1
$ cat <f1 <f3       # second < overrides first <

File created by >

$ ls >f2     # directory listing will contain f2
$ ls -l >f2      # f2 as empty 
$ >f3        # f3 created, empty.  No command executed.

Note: - is a synonym for standard input for some programs. Argument to these commands.

$ cat f1 - f2 >>fx          # from f1, then standard input, then f2

Note: /dev/stdin is a synonym for standard input in SVR4

$ cat f1 /dev/stdin f2 >> fx

Piping:

$ who >temp      #temporary file to hold output of who
$ sort temp

But can bypass this inelegant and inefficient use of temporary file by using a pipe (|) that connects the standard output of one program to the standard input of another program.

$ who | sort             | is the pipe operator.  creates a pipeline

Here, who's output is redirected (sent thru the pipe) to sort's input. Shell connects the programs together. Standard output of who is piped to standard input of sort.

$ ls -l | more      #page at a time directory listing
$ ls | wc -l

Programs in a pipeline (not limited to two programs, can have longer pipes, e.g. who|sort|lpr) run at the same time, not one after the other. Kernel looks after synchronization and buffering.

$ ps ax | wc -l
$ ps -aux | grep billybob | wc -l #number of processes billybob has
$ file /usr/bin/* | grep script      #which cmds in /usr/bin are scripts
$ cut -d: -f1 /etc/passwd | sort | more    #sorted list of usernames
$ wc * | tail -1

Powerful processing by connecting programs in a pipeline. Commands as single-step tools that can be joined together to do complex tasks. Pipe as the single most elegant mechanism in Unix.
Program by composition, not by custom-building. Reuse instead of re-create.

[?Circular pipeline? p1 pipes to p2 which pipes to p1. Not possible syntactically, but as programs...]

who | sort | lpr
(who | sort) | lpr
lpr (who | sort)

cat >f1 | cat
cat <f1 | cat >>f1

Remember:
redirection is with files
piping is with commands

  terminal (default)|                      |  terminal (default)
      or          
  file  <fn         | stdin COMMAND stdout |  file  > or >>
     or           
  pipe  |           |                      |  pipe   |