CMIS 325


David Wills


Know thy...
Commands intro
bash ease of use
File system
Shell benefits
Shell variables
Shell scripting

Unix Operating System

Operating system.

1. manages resources of computer system:
CPUs, memory, disks, devices, network, software/data (files)
Scheduling: who gets what when, how much and how long...

Efficiency--want to realize maximum potential of resources (minimize idle time, maximize throughput)
Fairness--at least don't starve anyone

Monitors, controls, mediates system usage.

2. provides user and application interface to hardware
makes computer easier to use, more productive
e.g. system calls (APIs), command language, utilities, GUI, VR...


Unix/Linux O.S.

Multiprogramming: several programs/processes/jobs can be in memory at once, in various stages of execution.

Multitasking: user can have several programs/processes/jobs going at once/in parallel/concurrently

Multiuser: Simultaneous use by many users. Non-interference with other users' activities and information.

Time-sharing: several users can use computer at same time, each thinking he is only user. Near illusion of being the only user. Interactive vs batch. "Time-sharing is a son of batch"

Multiprocessor: more than one CPU.

Multiplatform: x86, Sparc, Alpha, MIPS, PowerPC, S/390 (z Series), supercomputers, cluster...

Multi network protocols: TCP/IP, NetBeui/CIFS/SMB, Novell...
Multi filesystems: ext3, UFX, JFS, AFS, Reiser, FATs, NTFS, HPFS...

Real-time processing: at or within deadlines/intervals (predictable, bounded response time|). Some support (e.g. high priority given to real-time processes).

CPU switched several times per second among processes (implemented with timers that interrupt CPU periodically). CPU also interrupted by I/O devices, user processes requests for OS services, and user process errors.

Unix: early 70's developed at Bell Labs. Popular in universities. Much growth in 80's. Early PCs not powerful enough to run Unix; also, PC users didn't need multiuser, multitasking OS. Academic, research, engineering/scientific/manufactoring, DB and web servers, render farm. File, print, intranet, applications server. Workstation. Cluster.

Why was it successful?:

1.) Written in high-level language (i.e. C) vs. traditional OS much written in machine-specific assembly language (thus can only run on that hardware).
Portability: small modifications only are needed for it to run on different hardware. Linux on different processors: hardware independance promise of Unix realized?
Runs on full range of computers: single user micros to supers. "Mac II's to Cray II's". Update: Pentium II's to SP2. Update: P4 to HP's $24M supercomputer.
Linux can run on a 386, and versions bootable from a floppy (embedded OS), Live CD (bootable)
2.) Source code available for modification/customization/fixes. "open" system. Update: Open Source available to the world.
3.) Productive programming environment. (Not designed for data processing shops (characterized by large data transfer), or for use by receptionists, suits and homebodies).
-variety of small, powerful modular tools, easy to understand.
-tools can be easily combined to form new tools.
"power of a system comes more from the relationships among programs than from the programs themselves"
modular, tool-oriented building block approach to program design
Neither the most powerful nor the most efficient OS.
Lots of free software.
4.) Ease of networking.
DARPA support of BSD for communication.
Internet developed among Unix. TCP/IP native networking of Unix.
Distributed systems on different architectures facilitated by a common OS.
5.) troff enabled every fifth line numbering for ATT patent applications
"Unix": not an acronym, a pun on Multics: a large multiuser computer project of 60's and into 70's by Bell Labs, MIT, GE, then Honeywell (much used in Air Force), goal of a computer utility: all things to all people, hence large and complex. Unix was simple, i.e. "uni". Or a castrated Multics. Ideas from Multics: file system, shell as user process, separate process for each command.

Thompson and Ritchie (Turing award 1984, National Medal of Technology 1999) wanted an OS for minicomputer (DEC PDP-7, later PDP-11 [24KB memory, 512KB disk!]) that would facilitate cooperative programming and text manipulation utilities. Its coherent structure is due to its small size and design by only two people (the major users of the system, for their own covenience). Its reputation as cryptic and difficult to learn is absurd, eg. biff, the dog that barked. Unix is not chatty. "Unix may be difficult to learn but it's easy to use".

Design goals:
-simple and minimal function, not fast or sophisticated. User programs create sophistication.
-generality: single method serve a variety of purposes
--read/write files, devices, interprocess message buffer all with same system calls
--files, directories, devices: same naming, aliasing, protection
--processor traps and software interrupt mechanisms the same
-create environment in which large tasks are done by combining existing programs
Designed by programmers for programmers. Program development platform.
Networking and GUI accomodated although greatly increasing the size of the system.

Versions, vendors, vser groups, standards

Early versions/editions:
1 on PDP 7
2 on PDP 11/20
First Edition. Nov 1971
3(4?) in C, developed to support Unix, on PDP 11/70 with multiprogramming support
6 1976 first version widely available outside Bell Labs
7 ancestor of most modern Unices ~1979

Major historical flavors: AT&T System III and V (SVR4) and Berkeley BSD (defunct, whence csh, vi, TCP/IP and LAN networking, virtual memory). final BSD: 4.4 1993. /usr/ucb/bin for BSD commands on a SVR4 system
GNU ("GNU's not Unix") of Free Software Foundation: goal of independent and free Unix: emacs, bash, gcc, basic utilities, gimp, GNOME (but no kernel (HURD), yet)
GNU Public License (GPL): modifications plowed back into the source

USENIX users group
/usr/group (now UniForum
SAGE System Administrators Guild.

RHCE (RedHat)
Linux+ (
Brainbench, SAIR

Standards: what a Unix should look like, how it should behave.
Recommended books

Layers (concentric circles) of Unix: [hardware], kernel, utilities, shell , user.

Kernel: the operating system proper
--controls processes (creation, scheduling, deletion)
--controls/manages devices (i.e. I/O), CPU, and memory
--manages the file system
Accessible via ~200 system calls (requests for service, e.g. function calls in C to do device I/O, process creation/deletion, memory allocation/deallocation, file creation/access, signal handling). Different Unixes have compatible system calls but their implementation and system architecture (internals) can differ. To a program(mer) the system calls are the system. Shell and utilities use the system calls.
Kernel is the OS. All else is built on top of and from it. Loaded at boot time. /vmunix or /boot/vmlinuz file.
Original versions were 10,000 lines of C code with 1000 lines of assembly code for hardware functions unavailable or inconvenient in C. [Now? cf Multics, Windows]
Linux (strictly is only the kernel): highly customizable, can compile the source with the exact level and mix of support for the devices and services needed.
--command interpreter : prompts user, interprets (parses) the command line, arranges for command to be carried out (passes requests to kernel).
--a command line interface between user and kernel. (a GUI desktop environment (e.g. KDE, GNOME, Aqua) is also an interface between user and kernel)
--several available: sh (Bourne), csh (C shell and tcsh), ksh (Korn shell), bash (GNU), etc.
--programmable. Quick and easy programming (scripting), esp. for file manipulation (themselves and their contents). Can program on the command line.
--very flexible and thus powerful. vs GUI what-you-see-is-all-you-get
shell provides services to user. kernel provides services to programs, including the shell.
The different shells have different syntax. C shell has C-like syntax and better interaction. Bourne shell is the original. Korn shell and bash have C shell facilities but in Bourne shell syntax and are almost POSIX standard.
Reasons to know shell: 1. remote login via telnet or ssh. 2. no GUI available or needed. 3. fast 4. commands that have no GUI equivalent

Utilities: commands/tools/applications, e.g. compilers, editors, file manipulation, text formatters/typesetters, spreadsheets, DBMS, communications, servers, CAD, games, browser etc.
Users: programs become utilities. Extend Unix.
User programs not distinguished from system programs. eg. your own ls or more commands. [Exception are commands that require privilege, eg. passwd via setuid]

CPU in user mode (bit flag in processor) when shell, utility, or user program executing. Request for OS service issued with system call (e.g. functions available in C: software interrupt). CPU switches to kernel mode, performs the service, returns to caller in user mode. Certain operations, e.g. I/O instructions and system tables' access, only allowed when in kernel mode. I/O devices and timers can also signal request (interrupt) for kernel service. User mode processes can generate errors (traps) that cause switch to kernel mode; such errors include attempting to access another process's memory, accessing memory that is not resident in memory (page fault), processor errors such as divide by zero and illegal instruction.

Varieties of running Linux:

1. single floppy e.g. tomsrootboot
2. Live CD e.g. knoppix
3. Runlevel 3 (No X)
4. Runlevel 1 (single user) for maintenance
5. Runlevel 5 X (with or without a desktop environment)

Varieties of experiencing Linux:

0. install to disk, boot from it.
1. remote login (shell access) From Windows: telnet or putty ssh. Or from another Unix/Linux system using ssh.
2. remote X access from another Unix/Linux (run X clients remotely, display locally, login via XDMCP). From Windows runing an X server such as CygwinX.
3. On Windows, install Cygwin Unix environment. Install X server CygwinX.

X Window

from MIT's Project Athena, now X Consortium. X11 1987
XORG port of X to x86
X11R6 version 4 5?
(Mac Aqua is not X)

Windowing (ie. I/O) separate from interface (i.e window manager).
Client/server: client application does processing on a Unix system, the X server does the I/O (mouse and keyboard input, screen output) on an X terminal (could be over a network).
GUI is an application; can be run optionally or as needed.

X server:
  1. setup the graphical display (resolution, color depth, refresh rate) video driver.
  2. display windows
  3. track mouse and KBD activity
XFree86 is major X server for Linux.
X servers that run on MS Windows: MIX, xwin32, cygwin

Window manager (per server): special client that provides menus, icons, virtual desktop/workspace, window moving, sizing, iconifying, focusing, decorating (titles, buttons, borders).
Many window mangers available: twm, Motif mwn, Open Look olwm, fvwm, kwin, metacity, enlightenment... has them all.

KDE, GNOME, CDE "desktop environments": standardized look and feels and behaviors, drag and drop, set of compliant apps .

[display manager: gdm kdm xdm] | startx --> Xorg --> [desktop environment via e.g. gnome-session] --> window manager
X clients applications:
xlsfonts    list of installed fonts
xfd -fn     font viewer
showrgb     list of colornames
xterm      terminal emulator
startx initx  .xinitrc
Toolkits of window functions: Xt
Widget libraries: Athena, Motif, Xaw3D, Qt, gtk, Mesa(OpenGL), PEX

KDE: konqueror, Koffice, kmail
GNOME: galeon, nautilus, evolution
The Gimp: image manipulation "90% of PhotShop at 0% of the price"
ImageMagick: convert and manipulate graphics files in batches

Know thy multitasking, networked, multi-user, server system

lsb_release -a   #distro
uname -a   # lists name and version of operating system, architecture (CPU), IPC
arch      # architecture of system
hostname   # name of system    -i shows IP address
SuperProbe        # video card and its memory.  Linux
dmesg     # boot messages.  Linux.  hardware detected and drivers.
df     # filesystem(s) size.  -i shows inodes. -h human -T type
free      # how much memory 
top       # shows RAM
/proc/cpuinfo      # CPU type and current speed, cache size, BogoMips
/proc/interrupts   # IRQs and # of interrupts
/proc/devices    #files that have IRQs and other device parameters

GUI admin tools:
control-panel  Linux configuration: users, networking, printer, filesystem
linuxconf      admin configuration tool
webmin         web-based admin

who -i    # who's logged on, idle or active.  -l shows from where.
finger    # who's logged on
finger   #over network, if runs finger server.
w      # who's logged on.  Login time, from what remote host, idle time, running what command now, how much CPU time its taken (PCPU), how much total CPU time has used (JCPU).
last      # previous logins of users
lastlog   #last login of every user
/etc/passwd    #system users
write username      send msg 
talk username       chat

System state.
times     #bash builtin.  amount of CPU time used by shell and its processes
ps aux # processes in system: all users, in user format, also processes without terminal x
        $ -ef on SVR4
xload     # "load" of system: some combination of CPU and memory usage
xcpustate  # instantaneous percentage of CPU and memory usage, by categories
xsysinfo  #same info
xmeter    # various monitors (not in Linux?)  perfmeter
ifconfig      #network packets statistics
iwconfig      #wireless network packets statistics
netstat       #network connections.  -l listening ports. -t tcp statuses.
netstat -l    #ports being listened on
netstat -t    #active TCP ports
nmap          #port scanner
top        # Load average (average number of processes, i.e multitasking)
        # States of all processes: running, sleeping, stopped, undead
        # CPU usage: %user, %system, %idle
        # Memory: total , free, used, shared, buffers usage
        # Swap: total, free, used
        # per process information like in ps:  
        SIZE: virtual memory in KB
        RSS: % of physical memory
        WCHAN: system call sleeping on
        PAGEIN: #page faults

Intro to commands:

flexibility and extensibility of the command line. Not limited to GUI, can think outside the boxes with their what-you-see-is-all-you-get.
$ is shell's default prompt. Customizable by user (PS1 shell variable). > is secondary prompt (PS2).

Name of command as abbreviation or acronym. Command as a tool, easily combined with other commands e.g. in pipe or script. CLI (command line interface) scales well with experience, rewarding advanced users with greater productivity. CLI: continuous "programming" cmd by cmd. GUIs don't scale.
Every app can be started from the shell.
Bad news: commands are terse and non-interactive. Not known for hand-holding.
Good news: commands are terse and non-interactive
Unix is case sensitive.
pwd      present working directory.  where you're "at"
cd dirname       change present working directory to dirname
cd                change to home directory    
cd -              change to previous directory
ls       directory listing
ls -l      long listing gives more info about files
ls -a      show "dot" files, those that start with .
cat fn      display file 'fn'     
                Output can be paused with ^S, resumed with ^Q (on a
                slow terminal, or fumbled fingers) -n line numbers
more fn         display file 'fn' a page at a time.  
            Also: pg and less, head and tail
lpr fn       print file (or lp). to default printer. image and Postscript too.
rm fn        delete fn.  No recycle trash can.
rm f1 f2        delete f1 and f2
cp oldf newf     make a copy of oldf as newf
cp f1 f2 f3 dirname     copy f1, f2 and f3 to directory dirname
cp f1 .                 copy file to this directory
mv oldf targetf  move/rename oldf as targetf
mv oldf dir          move file into another directory
mv dir1 dir2         rename directory or move directory to be a subdirectory of another
mkdir dirname    make a directory
rmdir dirname    delete a directory (must be empty first)

wc fn            number of chars (bytes), "words", lines in fn -L longest line
sort fn          sort fn
grep word fn     search for word in fn
od -c fn         display ASCII chars of fn
file fn          what kind of file, e.g. C source code, executable
                 file, text file, shell script...
locate string         show all files in system that have string as part of name
find -name fn        search for file(s) in disk tree
who          who is logged on, on which terminals, doing what...
ps               what processes you have running in this shell.  Also, others'
                 processes  -e or -u user
ps ux                   all your processes in system, including x not
                           associated with this shell
ps aux                  all processes in system
kill pid     terminate a process
diff f1 f2       differences between f1 and f2
mail         send/receive mail:  Also mailx, elm, pine, mutt, GUI mailers etc.
vi           text editor
emacs        text editor. 
      Other editors: joe, jed, pico, xedit, kedit, gedit, kate (source code aware), kwrite
      Word processors: kword, abiword, wordperfect, openoffice writer
mdir a:             mtools to access FAT diskette
mcopy -t fil1 a:    copy file to floppy
mcopy -t a:fil2     copy from floppy
gcc prog1.c       C compiler.   executable in a.out
g++ prog1.cpp     C++ compiler
javac  Java compiler
java Hello        JVM
php, perl, awk, python, ruby, f77 compilers/interpreters
File manager/browser: mc, nautilus, konqueror

For more information on a command, in particular the options, use the man command:. Note: man pages are not meant to be drool-proof.
$ man grep
Each man entry is roughly:
Description (of options)
other misc. info, e.g. files used, bugs, authors, cross-references

[compressed troff/gnroff file in /usr/share/man/manN, uncompressed and formatted /var/cache/man/catN compressed text]

$ apropos string # search all the synopses for cmds with the string
$ whatis word
man intro #intro to Linux and shell

Unix Programmer's Manual (about 6 feet wide) had descriptions of how to use the utilities.

-h or --help option
bash help on its built-in commands

GNU info documentation on utilities /usr/info

HOWTOs and FAQs: /usr/share/doc
Linux Documentation Project:
comp.os.Linux newsgroup Linux Journal
Linux Magazine
Major utilities and distros have their own web sites: samba, apache, gimp, perl, sendmail, xfree86, gnu, gnome, kde, openoffice, applix, etc. has tons of the smaller open source projects.
Linux related web sites: end with .com or .org:
starting with "linux" or "li":
today mall links base now central app -center world


Categories of commands:
 Output of entire files::             cat tac nl od
 Formatting file contents::           fmt pr fold
 Output of parts of files::           head tail split csplit
 Summarizing files::                  wc sum cksum md5sum
 Operating on sorted files::          sort uniq comm ptx tsort
 Operating on fields within a line::  cut paste join
 Operating on characters::            tr expand unexpand
 Directory listing::                  ls dir vdir dircolors
 Basic operations::                   cp dd install mv rm shred
 Special file types::                 ln mkdir rmdir mkfifo mknod
 Changing file attributes::           chgrp chmod chown touch
 Disk usage::                         df du stat sync
 Printing text::                      echo printf yes
 Conditions::                         false true test expr
 Redirection::                        tee
 File name manipulation::             dirname basename pathchk
 Working context::                    pwd stty printenv tty
 User information::                   id logname whoami groups users who
 System context::                     date uname hostname
 Modified command invocation::        chroot env nice nohup su
 Process control::                    kill
 Delaying::                           sleep
 Numeric operations::                 factor seq
In detail:

Output of entire files
 cat               Concatenate and write files.
 tac               Concatenate and write files in reverse.
 nl                Number lines and write files.
 od                Write files in octal or other formats.

Formatting file contents
 fmt               Reformat paragraph text.
 pr                Paginate or columnate files for printing.
 fold              Wrap input lines to fit in specified width.

Output of parts of files
 head              Output the first part of files.
 tail              Output the last part of files.
 split             Split a file into fixed-size pieces.
 csplit            Split a file into context-determined pieces.

Summarizing files
 wc                Print newline, word, and byte counts.
 sum               Print checksum and block counts.
 cksum             Print CRC checksum and byte counts.
 md5sum            Print or check message-digests.

Operating on sorted files
 sort              Sort text files.
 uniq              Uniquify files.
 comm              Compare two sorted files line by line.
 ptx               Produce a permuted index of file contents.
 tsort             Topological sort.

Operating on fields within a line
 cut               Print selected parts of lines.
 paste             Merge lines of files.
 join              Join lines on a common field.

Operating on characters
 tr                Translate, squeeze, and/or delete characters.
 expand            Convert tabs to spaces.
 unexpand          Convert spaces to tabs.

Directory listing
 ls                List directory contents
 dir               Briefly list directory contents
 vdir              Verbosely list directory contents
 dircolors         Color setup for `ls'

Basic operations
 cp                Copy files and directories
 dd                Convert and copy a file
 install           Copy files and set attributes
 mv                Move (rename) files
 rm                Remove files or directories
 shred             Remove files more securely

Special file types
 link              Make a hard link via the link syscall
 ln                Make links between files
 mkdir             Make directories
 mkfifo            Make FIFOs (named pipes)
 mknod             Make block or character special files
 readlink          Print the referent of a symbolic link
 rmdir             Remove empty directories
 unlink            Remove files via unlink syscall

Changing file attributes
 chown             Change file owner and group
 chgrp             Change group ownership
 chmod             Change access permissions
 touch             Change file timestamps

Disk usage
 df                Report file system disk space usage
 du                Estimate file space usage
 stat              Report file or file system status
 sync              Synchronize data on disk with memory

Printing text
 echo              Print a line of text
 printf            Format and print data
 yes               Print a string until interrupted

 false               Do nothing, unsuccessfully
 true                Do nothing, successfully
 test                Check file types and compare values
 expr                Evaluate expressions

 tee                Redirect output to multiple files

File name manipulation
 basename           Strip directory and suffix from a file name
 dirname            Strip non-directory suffix from a file name
 pathchk            Check file name portability

Working context
 pwd                Print working directory
 stty               Print or change terminal characteristics
 printenv           Print all or some environment variables
 tty                Print file name of terminal on standard input

User information
 id                 Print user identity
 logname            Print current login name
 whoami             Print effective user ID
 groups             Print group names a user is in
 users              Print login names of users currently logged in
 who                Print who is currently logged in

System context
 date               Print or set system date and time
 uname              Print system information
 hostname           Print or set system name
 hostid             Print numeric host identifier.

Modified command invocation
 chroot             Run a command with a different root directory
 env                Run a command in a modified environment
 nice               Run a command with modified niceness
 nohup              Run a command immune to hangups
 su                 Run a command with substitute user and group ID

Process control
 kill               Sending a signal to processes.

 sleep              Delay for a specified time

Numeric operations
 factor             Print prime factors
 seq                Print numeric sequences
"Command" could be: program file (machine language compiled or a script), shell built-in command (statement), shell function, alias (whence -v or type to distinguish).
which cmdname to see if in system (actually, if in path)
whereis cmd tells where it and its man page is.
command format:
Consist of sequence of 'words' separated by white space (blanks and tabs).
A command can be one word: e.g.
A command can be more than one word. The second and all other words are arguments to the command: e.g.
  ls -l
  cp fromfile tofile
  grep word filename
  who am i
  nxterm -geometry 85x30+0+0 -fn 9x15bold -fg white -bg black -cr red &

Most commands look like:
cmd [options] [one or more filename(s) or words]

Not all commands have options and/or filenames. eg. date has no filename arguments, pwd has no options or filename arguments.
Options (flags, switches) usually specified by minus sign (sometimes +) and letter:
  ls -l
  ls -t
  ls -l -t
  ls -lt

more +500 fn start at line 500 of file

Some options followed by argument for that option:
  lpr -Pprinter4 fn

Each option meaningful only to the command:
 rm -i          interactively delete
 grep -i            ignore case in search

GNU commands also have "long" options: --
ls --color
easier to remember, easier to read in script, ran out of letters

System admin:

edit text configuration files, often heavily commented, e.g. /etc/passwd /etc/httpd/conf/httpd.conf
commands, e.g. adduser
GUI, e.g. system-config-users

Install software

RPM: RedHat Package Management

-qa          #list of all installed packages
--querytags  #  All possble info
-qa --qf "%{NAME} %{SIZE}\n"   #  Name and size of all packages

-qi package        #Info about this package
-qR  package       #what it requires
-qip package.rpm   #Info about this package in its file
-ql package        #All files of this package
-qf path-to-file   #what package this file belongs to

rpm -i arfarf-1.0-2.i586.rpm
installs the arfarf package of executable binaries, libraries,
documentation etc.
--force --nodeps  --test
-U package    #upgrade

-V package    #verify it's all there 
-e package    #uninstall (erase) it

/var/lib/rpm/  DB files

Desktop has  GUIs for software install.

yum does automatic download/update of RPMs. to 'find' 'RPM's on the 'net'

Debian Package Management

Ubuntu: uncomment lines in /etc/apt/sources.list to allow getting from 'universe' and 'multiverse' repositories
apt-get install packagename


-l              #list all packages
-L package      #files of package.   includes all directories in
-s package      #status
--print-avail package  #status
-S path-to-file #which package it belongs to
-I package.deb  #info about package file
-c package.deb  #files in the package file
-r package      #remove package (except config files)
  --purge package 


tarball: some software is distributed as "tarball", a compressed tar archive of source code files etc.
5 commands to unpack, compile, install:
1. tar -xzf tarballfilename.tar.gz #or .tgz Uncompress, Unpack. creates a subdirectory.
2. cd tarballdirectory
3. ./configure    #Generate customized makefile for the system.
4. make    #Compile using that makefile.
5. make install    #As root, install onto system.

Text editors in an xterm
vi (vim or elvis on some unixes) Unpleasant, but fast and ubiquitous. preferred for/by admin
"modal" :command mode and input mode

 x      delete character
 dd     delete line
 :wq    save changes and exit  : means an ed command
 :q!    quit, do not save changes
 i  go into input mode (insert)
 a  go into input mode (append)
 o  go into input mode (open new line)
If arrow keys don't move cursor: h i j k do.

Get out of input mode (back to command mode) with Esc.
Other useful vi commands (among hundreds):
:$      goto end of file
:r fil1     import file
:j      join two lines together
#yy     yank # lines to buffer
p       paste buffer contents
^f      ctrl f  scroll forward page
^b      scroll back
gvim X-enabled.
ed line editor.

the ne plus ultra of editors.
GUI, X-enabled
emacs -nw to run in an xterm
source code formatting "IDE"
extensible (has built-in Lisp system for creating commands, binding to keystrokes)
email reader, ange-ftp, news reader, file manager, run a shell inside emacs.
Always in input mode. Menu-driven commands (icons in xemacs), or keyboard are issued by holding Control or Alt.
Cursor movement by arrow keys, Home, End, Page Up, Page Down.
But if those keys don't work on your system (it's configurable):

up line      ^P  (means holding Ctrl, press lowercase p)
down line    ^N
beginning  line  ^A
end of line  ^E
down page    ^V
up page M-V   (means holding Alt, press lowercase v)
              M is 'meta' (emacs jargon). Might be press and release
             Esc on some sYstems.
Deleting text:
char        ^D or maybe Delete.  Backspace deletes char to left of
line        ^K
word        M-D 
A group of lines that was just deleted can be pasted by ^Y
Save changes     ^XS  holding Control, press x then s
Quit         ^XC
Load a file  ^XF  will be prompted for name of file
Insert a file   ^X i  (means do ^X followed by i, without holding ctrl)
Undo         ^X u
Cancel       ^G  to stop some inadvertant command
reformat paragraph   M-Q
switch to other buffer  ^X b
one buffer only         ^X 1
hundreds of other commands, including Eliza psychotherapist (M-X doctor)
Emacs configuration file: ~/.emacs
For a better looking emacs, edit your .Xdefaults file to change some of its emacs lines to:
emacs*geometry: 80x30+0+15
emacs*font: 9x15bold
Mail User Agents (read, compose, send email from mailbox) [vs Mail Transport Agent (mail server) that delivers, receives,and routes email]
mail also elm, pine, mutt, mh, xmail, netscape, thunderbird, kmail, evolution etc.
Send mail:
mail billybob
(type message here, end with . on line by itself, or by ^D)

Read mail:
  ? list of commands
  h list of messages, * is current message
  n display message #n
  r respond to current message
  d delete current message

bash Ease of use features

/etc/profile executed at login. system-wide set-up for all users
~./bash_profile or .profile executed. Terminal setup, environment variables (PATH, PS1),umask.
~./bashrc executed for each bash spawned. Aliases, functions and shell variables definition.
Previous pages of screen viewable by [Shift] PageUp or scroll bar
Mouse can drag copy (highlight) and paste (middle button) to command line or into application. Single, double, triple clicks for character, word, line highlight.
Linux virtual consoles. Alt F1, Alt F2 etc each a different terminal. X usually at Alt F7.
history of previous commands. Easily recalled and modified.
set history = 50 # here, the previous 50 cmds remembered.
History editing easier if arrow keys and editor (vi or emacs) cmds integrated as in bash and ksh (set -o vi).
Use emacs editing commands (65 of them!):
most useful:
Retrieve previous command with up arrow. Arrow keys move left and right in command line.
^A  beginning of line   Home key.
^E  end of line.            End key
^D  delete char.  Or Backspace to delete previous char.  Delete key.
^K  delete to end of line
^U  kill line (actually kernel)
Esc D  (or Alt D) delete word
!54  re-execute command #54
!la   re-execute most recent command that started with la
create new "command" by giving command another name.
alias newname=oldname
$ alias l='ls -la'
$ l
(output of ls -la here)

$ alias l='ls -la | more'
$ alias m=more
$ m myfile
(output of more here)
$ alias (to get list of all aliases )

alias 325='cd ~/umuc/classes/cmis325/students' #alias for a long path
alias all='ps -aux | more'
alias pt='enscript -B -h -f Times-Roman12 '

Typically aliases are in .bashrc so automatically available each login.
Unfortunately, bash aliases can't have arguments; but functions can.
h     history
lo    locate
j     jobs
cp    cp -i
mv    mv -i
m     more
g++   g++ -Wall -ansi
Filename completion service.
Shell will complete the name of a file. Activate with set filec, typically in .bashrc (or is part of emacs command line editing mode).
Only have to type a prefix of filename (enough to unambiguously specify the file).
$ ls
$ cat h<Tab> You type h and Tab key (or Esc twice), shell fills in rest of filename
(display of homeassgn1)

$ ls
$ cat h<Tab> since is ambiguous, shell beeps, cat home is on cmd line, waiting for
clarification, you type p<Tab>, shell completes to homeprob
(display of homeprob)
Also: completion of usernames, hostnames, shell variables, command names
bash: ^V to escape completion (when want literal tab)
Job control managing multiple programs from the shell
--switch jobs between foreground and background
--suspend a job, resume it later
If running X, not so useful (can have separate window for each program).
Stop the foreground job with ctrl-z (sends signal 18 SIGSTOP)
Run job in background with &

$ jobs
[2] Stopped vi novel
[3] - Stopped cc bigproc.c
[4] + Stopped vi bigtext
[5] Running prog1

+ marks most recently stopped job
- marks second most recently stopped job
Running is a background job

Resume most recently stopped job with fg
Resume any stopped job with fg %job_number, e.g. fg %2 or just %2
Resume a stopped job in the background with bg %job_number or %2 &
Stop a background job with stop %job_number
Terminate a job with kill %job_number
$ stty tostop #Background jobs will stop if they try to output to terminal.


Shell functions
--kept in memory (faster than script: No overhead of searching PATH, opening file from disk, spawning subshell)
--like a built-in command
--variables are global with shell unless typeset to create local variables
--private arguments (alias cannot have args in ksh/bash)
--define in environment file .bashrc, interactively, or in script
typeset -f to list them. or as part of output of set
declare -f
l() {
  ls -l $* | more

usage() {
  echo "Usage: et [-a name number | -d name | -p | name]"
  exit $1

e() {
  emacs -fg white -font 9x15bold -geometry 90x30+0+0 $* &

calc() {
  echo "$*" | bc -l
Create interactively:
function myfunname { cmds ;} *****************

Your terminal

Correcting typing mistakes: Erasing chars, killing (deleting) line.
In old Unix, # was the erase char: dd#ate == date
Now Backspace (BS) key, (same as ctrl h i e. ^H).
Can change so that any key is the erase char:
$ stty erase '#'      [quoting necessary?]
Change back to backspace char:
$ stty erase ' '   Backspace was pressed between the quotes, but since it doesn't have its  special meaning of delete anymore, it's just another character. Now BS is the erase char again.
$ stty erase ^h  Also set erase to backspace char.
$ stty -a         Shows signal mapping.  Can change interrupt, EOF, kill, word erase
Terminal physical set-up (local to terminal) and port set-up (what the kernel thinks the terminal is; stty or terminfo). Terminal is keyboard and monitor (or window system pseudo-terminal).
/etc/termcap: file of terminal descriptions/characteristics.
TERM shell variable shows terminal type. Some programs need to know about cursor positioning, scroll, reverse video etc.

Typed chars are read by the kernel (it does all I/O), erase, worderase (^W) and kill (usually ^U) chars are interpreted by it. If kernel reads the erase char, it discards the previous char and the erase char from its command line buffer. The special meaning of the erase char can be 'escaped' with the \ char. A \ followed by the erase char will not be erased, the \ will be discarded and the erase char will be treated as part of the command line. Upon entry of the end-of-line char (usually the Enter key), the chars comprising the command line are passed to the shell.
This interpretation of the chars by the kernel (more precisely, the device driver) can be turned off by putting the terminal in raw mode (needed by some full screen utilities). Usually the terminal is in cooked mode. stty sane might cure a psycho terminal. Control chars sent to terminal (e.g. display of binary file) can mess up terminal.
reset clears terminal if in "Martian" caused by display of binary file.
echo aBSm | od -c                 # m         
echo a\BSm | od -c               # a \b m    Escaping the erase char
stty command shows what the terminal looks like to the kernel. Interrupt (^C), quit, start and stop seem unescapable.

Original Unix terminals were teletypewriters (whence the tty nomenclature). Its control chars: backspace, tab, bell, CR-LF.
Return key (^M) converted to Newline (^J) in file, then to ^M^J upon display. (^J could be used in input instead of Return or ^M.)

^D is the conventional method of generating the end-of-file signal which terminates commands awaiting input from the terminal, e.g. shell. ^D send input to the program.

Unix has typeahead; don't have to wait for current command to finish before typing subsequent command lines. (for very fast typists or slow commands)

^S pause
^Q resume

Bell: xset b vol0-100 freqHz durationMs
echo -e \\a | od -td1

File system.

Files are the most important type of thing in a Unix system.
Provides a heirarchical (tree) naming structure for files (vs. traditional flat file structure per tape/disk). Start of the file system is the root directory, whose name is / (slash) [or it's called root and has no name]. Each directory contains the names of files and further (sub)directories. Directory is a file that contains information on how to find other files. Every file and directory has a parent directory (root is its own parent) Ordinary files are leaves of the tree.

The pathname is the sequence of directories that the file is in, starting with /. Absolute file name is unique in the system.

File is a named place to store information. A Unix file is (only) a sequence of chars (bytes): no formatting of file contents is provided by Unix. This is simple and uniform, there are no multiple file formats that Unix has to deal with. File is independant of an application. Any internal structure or format of a file is the responsiblity of the program or application that uses the file. No concept of records, no access methods. File can be accessed sequentially and randomly. File contains whatever user puts into it; no structure except what the user imposes. Not even the newline char meaning the end of a line is known by the kernel (except device driver).
Executable files are identified by permissions, not by name (e.g. DOS extension).

Chars can flow from one program to another; simplicity of interconnection. A common interface. Programs can pass their output directly to other programs as their input. Programs expect a stream of chars as input and send a stream of chars as output.
There is no end-of-file marker in a file.
"File" as source of input or target of output, eg. devices are files in Unix.

File and directory names are limited to 255 chars. Any char except / and null character, (includes (invisible) control chars, space, newline), can be part of a filename, e.g. f!@#$%^&*() is a valid, albeit undesirable, filename. Novices somehow manage to create such filenames.
Each user has a home directory assigned by Sys Admin, specified in password file (/etc/passwd); is the pwd upon login. ~username equals the pathname of the user's home directory.
A logged-on user always has a current working directory; the directory the user is "in". An attribute of the process. Filenames specified are relative to the pwd (unless start with /, then is absolute).

df: info about filesystems. Availability, capacity, percent used, number of files. -i inodes -T type
du: info about disk usage. Number of blocks allocated to files. du -s for summary.
Hard links (to files only) create DAG. Symbolic/Soft links to ancestor directory creates cycle: subdir B of A has link in it that links to A.
find command: search "database" of filenames: owner, size, timestamps etc.
find / -size +1000 -mtime +30 -exec ls -l {}\; #does ls -l on files >1MB unmodifed in 30 days
find / -type f \(-perm -2000 -o -perm -4000 \) -print #finds setuid and sgid files
fuser shows user of a file
quota to limit disk space used by a user
ulimit -f maximum file size user can create


Shell: benefits

general-purpose application. An app for running apps.
1. generate filenames from a user-specified pattern (wildcards).
2. input and output can be redirected to/from files and programs, not just terminal.
3. create your own customized environment (shell options, aliases, variables, scripts, functions)
4. programming on the command line
Shell is a program that prompts for a command line, interprets it, runs program.

shell metacharacters (operators of the shell language)

Filename generation: (globbing) match existing filenames

  *  match any string of chars, including null string.
  ?  match any single char.        * and ? are called wildcards.
 []  match any one of the chars enclosed.  A pair of chars,
  separated by a minus will match any char in ASCII order. [abc] [a-z]
  [!]  or [^] match any char except one of the enclosed. [^a-z]
Generated filenames are in ASCII order.

Example patterns:
*     all files
f*    all files in current directory that start with f, including f
*f    all files that end with f, incl. f
f?*   all files in current directory that start with f and have at
         least one other char
[a-z]*        start with a thru z
[a-z]         single lowercase letter filenames
f[aeiou]?  start with f, followed by a vowel, followed by one char
*.gif          all files ending in .gif
*.???         all files ending in a DOS-like extension
/etc/rc5.d/S*  files in that directory that start with S
Ex. commands
$ ls -l *.cpp           #files that end in .cpp
$ grep rand *.cpp   #search for rand in all C++ programs
$ ls -l [a-z]*[0-9] #lists files that start with lower letter, end with digit, anything, or nothing, between.
$ rm *[0-9][0-9]    #deletes all files whose names end in two digits

$ rm *[10-99]       Warning: shell doesn't know numbers.  This is not ten thru ninety-nine.  1 or 0 thru 9 or 9 (poorly written, is equivalent to [0-9]).   ASCII sequence only.

$ rm * [0-9][0-9]       
Warning: space after *, will match all files and delete them all!
Exception: * won't list files that start with . (dotfiles, so-called hidden files). Must do .* to generate their names. Such files are not usually listed by ls (use -a option of ls); mostly are configuration files (customizable startup) for various apps and shell.

Every directory has these two filenames (links): [to access their inodes]
. this directory. [so can open cwd without knowing its pathname]
.. its parent directory

Exs. file1 == ./file1
$ cp /home/billybob/file1 .       #copy into current directory
$ cd ..          # move up a directory
$ cp ../file .    #copy file in parent directory to this directory
$ mv ../*.h .  #move the .h files from parent directory to this one
$ mv *.h ..        #move them back up

echo command displays its arguments. Useful for seeing (and learning) shell's interpretation of command line. also to display value of shell variables and as print statement of scripts.
$ echo hello world
hello world
$ echo hello           world      #2 arguments
hello world
$ echo f*                # shell expands to list of all matching files
f1 fil2 fred.c         # all the filenames that start with f 
$ echo *                # all filenames
f1 fil2 fred.c mystuff junk 
$ echo *[0-9][0-9]
$ echo asdfasdf >afile

If no filenames match the pattern, the shell passes the pattern itself:
$ echo f*5
$ echo *     #in an empty directory

Command line is given to shell, shell sees * or other filename matching chars, expands to the list of files, then gives the list to the command as arguments. The command never sees the pattern matching chars! The pattern matching chars are not arguments to the command.
User command line   After shell processing
    ls f*       ls f1 fil2 fred.c
    f*      f1 fil2 fred.c   
                      (probably error, would try to execute command f1)
Shell order of evaluation of command line: simplified
  1. keywords (if, case, for, function, while, !, {} ...)
  2. aliases
  3. built-in commands (alias, cd, echo, export, history, jobs, kill, read, set, shift, test, trap, umask ...)
  4. functions
  5. PATH search for command
Shell options. set -o list on/off values **************************************************
Input-output redirection.
Most commands can take input from standard input, and send output to standard output. Defaults are keyboard and monitor, i.e. the terminal [device file of terminal]. But can change what the standard input and standard output are.
standard error is where errors/warnings sent; default is terminal, i.e. same default as standard output. I/O "instructions":
> redirect output to file (Warning: wipes out current contents of file, if any, unless noclobber is on).
< redirect input to come from file
>> redirect output (append) to file
2> redirect standard error to file
AKA stdin, stdout, stderr
File descriptors: 0 stdin, 1 stdout, 2 stderr
$ cat f1 >f2         # copy f1 to f2
$ cat f3 >>f2     # append f3 onto f2
$ cat <f1            # display f1
$ cat f1 f2 f3 >f4   # all to f4
$ cat f1 f2 f3 >>f4   # all appended to f4
$ cat f1 >dir1       # ERROR.
$ mail billybob <letter3 # letter3 is the message
$ cat f1 >f1         # wipes out f1
$ cat <f1 >f1     # Warning.  creation of f1 wipes out contents
$ cat f1 f2 >f2      # net result is copy of f1 in f2
$ cat f1 >>f1 # not allowed: won't append f1 to itself  
                [if f1 not exist creates empty]
$ cat <f1 >>f1     # not allowed.  recurses?!
$cat f* >f5
$ sort xfile >xfilesorted   
$ grep math *.cpp >mf   #all lines containing 'math' from all the 
                    C++ files into file mf
$ file * >filetypes     #type of all files
 cat f1     filename argument
 cat <f1 redirect input so comes from file f1, cat command does 
        not see redirection. No file argument for cat command here.
 cat >f2     output redirected to file f2.  No file arg to cat.
  What's the input? Keyboard or mouse paste.  Will accept input until ^D.
If no filenames specified to command, will take input from standard input (allowing input redirection and piping).

Shell does the redirection, command program is oblivious as to where standard input is coming from (terminal or redirected file) and to where standard output is going to (terminal or file).

Shell doesn't care about format. Redirection symbols and filenames removed from command line.
$ >f2 cat<f1
$ cat <f1 <f3       # second < overrides first <
File (re)created by >
$ ls >f2     # directory listing will contain f2
$ ls -l >f2      # f2 as empty 
$ >f3        # f3 created, empty.  No command executed.
Note: - is a synonym for standard input for some programs. Argument to these commands.
$ cat f1 - f2 >>fx          # from f1, then standard input, then f2
Note: /dev/stdin is a synonym for standard input in SVR4
$ cat f1 /dev/stdin f2 >> fx


$ who >temp      #temporary file to hold output of who
$ sort temp      
But can bypass this inelegant and inefficient use of temporary file by using a pipe (|) that connects the standard output of one program to the standard input of another program.
$ who | sort             | is the pipe operator.  creates a pipeline
Here, who's output is redirected (sent thru the pipe) to sort's input. Shell connects the programs together. Standard output of who is piped to standard input of sort.
$ ls -l | more      #page at a time directory listing
$ ls | wc -l
Programs in a pipeline (not limited to two programs, can have longer pipes, e.g. who|sort|lpr) run at the same time, not one after the other. Kernel looks after synchronization and buffering.
$ ps ax | wc -l
$ ps -aux | grep billybob | wc -l #number of processes billybob has
$ file /usr/bin/* | grep script      #which cmds in /usr/bin are scripts
$ cut -d: -f1 /etc/passwd | sort | more    #sorted list of usernames
$ wc * | tail -1
Powerful processing by connecting programs in a pipeline. Commands as single-step tools that can be joined together to do complex tasks. Pipe as the single most elegant mechanism in Unix.
Program by composition, not by custom-building. Reuse instead of re-create.

[?Circular pipeline? p1 pipes to p2 which pipes to p1. Not possible syntactically, but as programs...]
who | sort | lpr
(who | sort) | lpr
lpr (who | sort)

cat >f1 | cat
cat <f1 | cat >>f1
redirection is with files
piping is with commands
  terminal (default)|                      |  terminal (default)
  file  <fn         | stdin COMMAND stdout |  file  > or >>
  pipe  |           |                      |  pipe   |



Files and processes are the main things in Unix system.
A program/command when started/invoked/run becomes a process in the system. A process is created for every invocation of a command/program. (Exception: "commands" that are built in to the shell, e.g. cd, expr, test, trap, exit, control constructs, see man page of shells or bash help for the built-in cmds). [Another exception: dot command runs a script in the same process as the shell]

A process is an executing program. Shell spawns a process to run each program/cmd. Kernel manages all processes (creation/deletion, CPU scheduling). Scheduled by kernel (ie. which process gets CPU and for how long). Objective of Unix scheduling is responsiveness for interactive users. Long running processes have priority lowered. System processes have higher priority than user processes.

Process has its own memory address space, owner, parent, process id (PID), cwd, and state (running, waiting for I/O, ready/sleeping). Process Control Block (PCB) and other kernel data structures track processes.

Process can communicate with others, e.g. pipes. Other IPC mechanisms in programs

ps command lists your processes under your terminal:
$ ps
2168   u05     S        0:03    bash
2263   u05     R     0:00     ps
This user has two processes; one, with PID 2168 is the sleeping shell process of the user, the other with PID 2268 is the running process that is the executing ps command.
R running or ready
S sleeping
I idle?
Z zombie
D disk wait
P page wait
W swapped out
N lowered nice priority
T terminated

ps -e   all processes in system.     or aux
ps -f   more info: when started, parent process PPID
ps -l   more info: priority, memory used, state
Process has an environment defined by various system and user-created variables. Inherited from parent.

When using a pipeline, each command in it has a process that is the running command.

Background processes.
$ mozilla &
$ gcc pgm1.c &
& tells the shell to run this process in the background, the terminal will not be tied up while the command executes, the prompt will be displayed, further commands can be entered. Useful for non-interactive ("unattended") jobs that take a while. The gcc process is running in the background; when it completes, a message will be displayed. The shell and gcc processes are running concurrently. Background jobs can send output to terminal [unless stty tostop] but can't read from the terminal. Only the single foreground process can read from the terminal. Can have many background processes simultaneously.
$ sort bigfile | lpr & #entire pipeline run in background.  | higher precedence than & 

Processes are in a tree-like heirarchical structure. You have a shell process running that is interpreting your command lines and invoking other commands/utilities as requested. Each invoked program is a child process of the shell (and the shell is its parent). Booting loads kernel (/vmunix or /boot/vmlinuz) into memory; it initializes device drivers; it creates process 0 (swapper) or sched which in turn creates process 1 (init) which spawns terminal port getty processes [or ttymon process] which wait for login name, then passes it to a login process which validates user and then replaces itself with the user's login shell process.
init also runs the /etc/rc files, brings system to runlevel and starts all other system processes and daemons (via their scripts in /etc/init.d/).
/etc/inittab "runlevel" of system, i.e. single-user 1, multi-user 3, X 5

Parent forks child and waits (unless child as background process). Child execves, runs, exits.

If an ancestor process is destroyed or terminated, all descendants are sought out by the kernel (based on info it maintains about all processes, in particular the parent-child relationship) and are terminated.

$ kill -9 pid
# terminates process with PID of pid. -9 immediately. Useful for runaway processes, eg. infinite looping, terminal lockup. Rare that user software locks up computer. Rare that Unix crashes or needs to be rebooted.

$ kill 0
# terminate all processes in process group (all descendants of this shell, e.g. all background processes)

$ kill -9 0
# terminate all descendants and self

Child process that terminates and release resources notifies its parent and waits for reply. Is a defunct/zombie process until gets the acknowledgement from parent. If parent of a defunct process exits before the defunct child, the defunct child is adopted by init and becomes a zombie process (usually removed at next boot and doesn't affect performance).
Funny state: process forgets what shell invoked it, or what terminal it's connected to.
[If a script that starts a background job terminates before the background job, the background job becomes an orphan.]
Killing a process in a pipeline will terminate all the other processes in the pipeline. Processes in a pipeline are in the same process group.
ulimit -u maximum number of processes creatable by user

Sequential commands.
$ date; who ;pr fn 
sequential command: semicolon is shell meta-character to allow multiple commands on one line. The commands are executed in sequence, just as if they were on separate command lines. Useful for putting several commands in background or at or mulitline command on one line:
$ for i in `cat classlist` ;do mkdir $i ;done
syntax requires that do and done must be at start of line

$ (sleep 300; echo Wake up) &    # catnap for 5 mins.   
                    & higher precedence than ; so need parens

Subshell spawned.
(date;cat f1) | lpr

[Parenthesized cmds are run in their own subshell. ksh]
$ wait
until all background proceses are done. useful to synchronize in script

$ nohup command &
Keeps a background command from being terminated when logout (all user processes terminated when logout). The command is a descendant of your shell but is not destroyed when your shell is terminated upon logout. Output to nohup.out, unless redirected. Note: csh background processes are not terminated upon logout. If parent of a background process is terminated, init (PID 1) becomes parent.

$ time cmd #run the cmd and also report how much CPU and wallclock times

$ nice -increment command
Lowers priority of command.. Processes in system compete for resources, eg. CPU, memory, disk swapping. A long running process can have priority lowered. Kernel dynamically recomputes priorities, usually to favor interactive processes so users get better response. Typically on Unix: lots of small interactive jobs. CPU intensive processes get priority lowered.
Linux priority: -20 to 20 (high to low), 0 is base

$ batch cmd
# system chooses when to run command
A true batch subsystem as found on mainframes supports multiple queues, with in-queue priorities, queue execution priorities, resource limits, permissions. NQS
$ at time command        Runs the command at the specified time.
                       E.g. start it at midnight 
$ at -f script-file 4 am friday
$ at -l         # list my at jobs
$ at -r id      # remove an at job

$ crontab file-of-cmds-and-times
Periodically run cmds specified in file. System administration: backups, cleanups, accounting.

at and cron if have permission in /etc/cron.d/at.allow and cron.allow

Daemons: server processes not connected to terminal; usually started at boot, wait in background for request for service. E.g. crond and atd wakes up periodically (once a minute?) and executes jobs scheduled for that time. System daemons for administration. lpd/cupsd printer daemon, syslogd and klogd accounting, xinetd TCP services, updated flush disk buffers in memory to disk, httpd web server, named DNS server, sendmail mail server, nfsd NFS services, smbd Samba, nmbd NetBuei names

Multithreaded execution: process composed of threads that share the same address space. Less overhead than multiple processes.

Process's virtual address space divided into same-sized pages, each can be loaded into any same-sized page frame of physical storage. If memory heavily allocated, thrashing occurs as page faults increase so that little work gets done; then an entire process will be swapped to disk.
Shell special chars "metacharacters" (so far):
* ? < > | & ; [ ]
You are giving these to the shell, not to the command.

Another one: ` backquote char, always used in pair. Command substitution: replace with result (output) of running the command. Replace command with its output.
$ date
Tue  12 Apr  12:20  1995
$ echo date
$ echo `date`
Tue  12 Apr  12:21  1995
$ echo Today is `date`
Today is Tue  12 Apr  12:21  1995
$ `date`       is replaced by its output, attempt is made to execute
Tue: command not found

$ ls -l `which cat`    
#long listing of the cat command file, wherever it is

Useful to give list of filenames to command:

$ wc `locate .java | grep wills`  #wc of wills' .java files

$ ls -l `find . -name 'core*'`  # listing of all core dump files

$ file `locate .gif` | egrep -v 'GIF|symbolic'  
#any .gif filename that is not a GIF file nor a symbolic link

Which bash builtins are also the name of commands in your PATH of directories?
$ for i in `cat bash_builtins` ;do which $i 2>/dev/null ;done

$ mail billybob barack hillary rudy mitt <letter1
$ cat >tolist
$ mail `cat tolist` <letter1

$ cat  alf
$ echo `cat alf`                # echo turns newline into space
a b c d

New syntax for command substitution: $(cmd) Allows nested substitutions.
$ echo $(date)

Ex. listing of README files that are larger than the README file in this directory:
$ ls -l $(find . -name README -size +$(echo $(cat ./README |wc -c)c)-print)
$ ps -aux | fgrep "`who | cut -f1 -d' '`"    
#processes of logged on users
Escaping shell metacharacters:
Can turn off special meaning of shell metacharacters by "escaping" them. One way is by preceding them with \ escape char. Metacharacter is then treated like ordinary char, its special meaning to shell is escaped; backslash discarded.
 echo *         all filenames
 echo \*        * 
 echo \x        x 
 echo \*\?      *?
 echo \\        escape the escape char, so \  is echoed
 echo ab\BScd   escape the backspace char.  acd on terminal, 
                but ab\bcd
 echo ab\BSBScd backspace over the escaped backspace.  
                abcd on terminal
 echo \BS       is \b
(Pipe these into od -c to see the chars:  \b is backspace).

Escaping also with pair of single quotes (apostrophe):
Quoting turns off special meaning of shell metacharacters. Shell strips off the quotes, then gives to command. Preserves whitespace. Quoted string is one argument.
echo '*'        *
echo '*?'       *?
echo '\*\\'     \*\\
echo 'hello'    hello
echo hi   bob   hi bob            #two args to echo
echo 'hi   bob' hi   bob      #one arg to echo
echo 'abBScd'   acd   Kernel does erase processing
echo 'ab\BScd'  echoes acd, but is ab\bcd   Kernel-escaped the erase

If a filename is *, can cat that file: create it with >'*'
$ cat \*
$ cat '*'

shell secondary prompt PS2 for incomplete command line, > by default
$ echo 'hello
$ echo hello\   #\ at end of line means continued on next line.  newline effectively ignored.

Try echoing:
\\\$                     \$
\\\                       secondary prompt 
\\\\                       \
\\ \\                      \ \
\\\\\\                      \\

echo escape sequences: \b backspace, \f formfeed, \n newline, \c no newline, \t tab, \\ backslash, \nnn octal
$ echo -e 'hello\b wor\nld \t stuff\c'  # quote to protect from shell
hell wor
ld  stuff$

Linux bash:
$ echo -e                   #to enable the escape sequences
$ echo -n hello world       #suppress newline

File creation.

1.) kernel allocates space in storage (disk) to hold the file. Text is stored one byte per char. Non-text files (e.g. executable object code) allocated number of bytes needed. File holds the chars (contents of the file), nothing more. No end of file marker, does not have its own length, permissions, owner, etc.
Storage allocation in blocks (512 bytes or 1024 , 4K sys dependant?]

2.) inode is created for the file in inode area of disk. Inode number is index into inode area. (e.g. inode number of / is 2).
--type of file (ordinary/regular, directory, device (special) character or block, symbolic link, socket, fifo pipe)
--location of file contents on disk (15 pointers: 12 to direct blocks, 1 single indirect block, 1 double indirect block, 1 triple indirect block)
--size of file, i.e. number of bytes
--time of last modification
--time of last "access" (execution or read)
--time of last change of inode
--permission bits
--owner uid
--group gid
--number of links to this file (i.e. number of filenames that are this file)
$ ls -i      # shows inode numbers of files
$ df -i     # inodes per filesystem 
Each inode entry has unique inode number. Thus each file has unique inode number. Unix uses the inode number to refer to the file. The file name is only a convenience to us. / is inode 2

3.) Directory entry created for the file. Obsolete: Directory is a file of pairs: inode number (2 bytes), file name (14 bytes). Now is variable length triples: size of entry, filename, inode number.
Only kernel can change directory contents. Directory file could be read to see inumbers and filenames: od -c

Picture of file/inode...

Contents of file in one place (actually could be scattered in several blocks on disk ; larger the file, the more blocks). Most of file info in another place (i.e. inode). Name of file in another place (i.e. directory). Filename to inode number to the inode entry to data blocks.

User issues: cat f1
Each char collected by kernel's terminal I/O (device driver). Erasing handled by kernel, shell does not get erase char. Upon newline char, kernel passes line to user's shell process, shell starts ("spawns") a cat command process as a child process and goes to sleep (waits until child process has finished, or died). cat process has the kernel read current directory to find the inode number of file f1. Kernel then finds the inode entry in the inode area of storage to get the address of the start of the file's contents and if the permissions allow the user to read f1 then transmits the contents of the file to the user's terminal. It transmits exactly the number of chars indicated in the inode entry as the size of the file. The cat process terminates, the shell is awoken and prompts the user (via the kernel) for the next command.

Inode and inode number of a file and contents of file do not change if the file is renamed or moved to another directory, only directory entries are changed.
Copying a file creates a new file, i.e. a separate inode and contents area.
Rename: change directory entry filename.
Move: remove directory entry for this file, add it to the new directory.
Move a directory: remove its entry in parent's directory file, add entry to new directory
[move to descendant?]


Many names for one file (inode) with the ln command (directory entries that have the same inode number). File can be known by more than one name in one or more directories. File can be "in" many directories (but only within same filesystem).
Saves storage of duplicate files; centralized copy. Convenience of multiple names.
$ ln f1 linkname
Makes a directory entry with filename linkname with same inode number as f1. Two names for the same file. Can remove one without deleting the file. inode entry keeps count of links, file only deleted (inode entry and storage freed) when all links gone.
Hard link: no difference between original and the link; both are "links" to the file/inode.
rm removes a link (deletes file only if is last/only link to the file).

$ ls -l
shows number of links each file (i.e. inode) has and whether filename is a softlink.
Copying a file makes a copy of file contents and a new inode entry (with a different inode number).
Moving a file only changes directory entries; file contents are not moved (unless moved to another filesystem [eg. partition]).
Change permission on link changes all links' permissions.

Symbolic link (soft link/ symlink) (ln -s) is a file containing pathname of another file (MS: shortcut). Separate inode for a soft link. May point to directory. ls -l shows 'l' as first char of permissions of soft link file; and object pointed to.
Can soft link across filesystems.
ln -s existing [symlinkfile]
Useful for
1. compatibility: one system can have directories or files in different places but links make it look like other system.
/etc/X11 -> /usr/X11R6/lib
/etc/grub.conf -> /boot/grub/grub.conf
/usr/tmp -> /tmp
2. multiple names for same program. e.g. fgrep egrep are links to grep (will know how called and acts accordingly).
3. make a distant part of file system close:
ln -s /media/hda2/home/wills fedora5

dangling link: blinks red

Hard links make filesystem a DAG (directed acyclic graph). Soft links can make it a graph (eg. cycle if link to an ancestor directory).

File security

User can control who can use his files. Unix divides all users of a file into three categories: user (owner), a group the file is in, all others (world). Kernel enforces security, not the shell.

Group names and members are defined by Sys Admin in /etc/group. User can be a member of several groups. A file is in one group (can change which group a file is: chgrp). Change owner of file with chown (unless diabled). id to show current group. groups to show groups you are a member of. newgrp to change current group (SVR4), or are simultaneously in all groups, created file given group of directory it's in (BSD).

Standard Unix: No access control lists ("discretionary" security) [AIX has ACLs]. No security classifications ("mandatory" security) except see
NSA's Security-Enhanced Linux. No auditing of file accesses (except times of last modification and access).

File permissions:
For each category of user relative to the file, there are 3 types of permissions on a file:
Ex. myfile
user:   rwx Owner has all permissions.               
group:  r-x Group has read and execute permissions.               
others: r-- Others can read the file.
Permissions stored in inode.
$ ls -l myfile
-rwxr-xr--   1  billybob  cmis325     261   Jan 16 12:32  myfile 
Change permissions with chmod: only the owner can change the permissions
u=user, g=group, o=others, a=all: +/- r, w, x (s for SUID, SGID)
$ chmod u-x myfile   remove user's execute permission 
$ chmod +x myfile    add execute permission to all categories of users
absolute permissions: specify all bits' values with 3 octal digits:
r:  444 
w:  222 
x:  111 
$ chmod 751 myfile        
        111  -rwxr-x--x

-rw-------   is 600 owner r and w
-rwx------   is 700 owner rwx
-rw-r-----   is 640 owner r and w, group r
-rw-r--r--   is 644 owner r and w, all others r only
-rwxr-x---   is 750 owner rwx, group rx
-rwxr-xr-x   is 755 owner rwx, group and world rx

Directory permissions:
r: read, e.g. ls. Read the directory file (pairs of filename/inode number). Can know the names of files in the directory.
w: write: create and delete files in the directory. Can delete or move a file even if don't have write permission on file itself.
x: search or traverse directory for a file to use, execute a file in the directory, e.g. cat file, cd to a directory. ls -l
Need w and x on directory to delete a file in it, even if can't see it.

r-- can see what files are in the directory, but can't use them
--x can use a file that you know is in the directory
web pages only need be r for web server, directory needs only x?

Need search permissions on all directories in path to file.

A directory will have first char of ls -l be d
$ ls -l
drwxr-x---   2  billybob    cmis325      512 Sep 25 13:49 subdir1
Here, owner can do anything in subdir1, group can use the files there and see which files are there, world has no access to files in subdir1.
$ ls -lai /  # / is inode 2.   . and .. are same, / is its own parent.
Sys Admin (superuser, root) not constrained by permissions, has all access to all files. Can do anything with any file or directory. Except constrained by permissions on files owned by self, e.g. /etc/shadow.

Login name is actually a userid number. root is userid 0.

/etc/passwd is list of all users of system:
name:encrypted password:userid#:group id#:Name:home directory:shell
Or encrypted password is in shadow file: and password field in passwd is x
/etc/shadow is list of users and their encrypted passwords. Only root can read.
name:encrypted password:password aging fields

umask specifies (reverse) default mode for created files
Default is 666 for files, 777 for directories. Security concern if account setup with the defaults.
/etc/profile always executes before user logged; usually has umask changed.
$ umask         # shows umask setting
$ umask 022     # files will be 644, directories 755
$ umask 077     # files will be 600, directories 700
Setuid permission bit: US patent 4,135,240


--viewed as files.
--are filenames in /dev. Known to the kernel as device interfaces but are accessed by user's process the same as regular files.
Device-independant I/O.
OS hides the peculiarities of hardware devices from program. Program accesses device as if it were a file.

To a program/cmd, every device is the same, i.e. a file. Cmds aren't concerned with devices, they don't need to know anything about devices (how to operate them).
When command accesses one of these "files" the kernel does all I/O so only kernel need handle the device characteristics. Kernel has device drivers to interface with and control peripherals. Device and its controller interface with device driver. Requests to access device file causes activation of its driver.
Devices mapped into the filesystem, simplfying program's interface to them (to a program a device is just a file). Gives user a consistent mechanism of interaction.
$ ls -l  /dev
crw--w----  1  billybob   tty           24,2     ....      tty01
brw--w----  1  root       sys            6,1                 mt0
No data blocks allocated to these files.

c: character device (terminal, printer, modem)
b: block device (tape, disk). Use block buffer cache for efficiency.

major and minor numbers : major is device type (ie. driver), minor is instance (particular physical unit)
/proc/devices is a list of them
Kernel uses major number to redirect I/O request to appropriate driver. Driver uses minor number to know which specific device to access.

execute status always - (devices are only read and written)
/dev/lp0    DOS LPT1:
/dev/lp1    DOS LPT2:
/dev/ttyS0  DOS COM1: 
/dev/cua0   DOS COM1: old kernels
/dev/mouse  link to mouse port
/dev/fd0    floppy A:
/dev/hda    IDE primary master
/dev/hda1   first partition 
/dev/hdb    IDE primary slave
/dev/hdc    IDE secondary master
/dev/hdd    IDE secondary slave
/dev/sda    SCSI drive or emulated USB, SATA
When using terminal, user is owner of it. Usually only owner can read it, otherwise someone else could "steal" your input. Can allow anyone to write to your terminal (their output would intermingle with yours). [2 writing to a file at once]

tty to see what your terminal name is.
$ ls -l `tty`       # to see your terminal device file
/dev/tty synonym for terminal you are using [same for all users, so how: kernel?]
ls > /dev/tty
ls > /dev/tty01
All 3 have output at your terminal (assuming terminal is tty01).

cat </dev/tty
cat </dev/tty01
cat </dev/tty02 # if have read permission on tty02 can steal its input.

Useful if program has input and output redirected but needs interaction with user at terminal:
$ crypt <file >e_file   #crypt is doing: echo "Enter key:\c" >/dev/tty
Enter key:
/dev/null "bit bucket" , throw away output, endless sink.
$ time sort bigfile >/dev/null  #if only interested in time to sort.
                      output of sort discarded.
Inputting from /dev/null is end-of-file immediately.
$ cp /dev/null f1 #zeroes the file f1 but leaves other inode info the same
/dev/zero endless source of NULLs


partitioned into subsystems (i.e. disks and partitions), all in the Filesystem of /.
Each called a filesystem. Each has own set of inodes. Can't hard link across filesystems (this is the only aspect about multiple filesystems that affects average user). Also each filesystem must be mounted to be accessible; like grafting onto the tree. E.g. filesystem for projects, facilitate backups (mostly only user directories are changing), reduced head contention.

/etc/fstab list of filesystems
/etc/mtab mounted filesystems

df to see mounted filesystems and amount of storage (number of blocks) and inodes allocated and free.

Each filesystem:
block 0 is boot block, used when booting system. Unused except on root filesystem.
block 1 is superblock: size of filesystem (fsize), size of inodes area (isize), block size
blocks 2 to isize+2: inodes (8 per block)
blocks isize+2 to fsize-1: contents of files (incl. indirect blocks) and free blocks

logical entire File system---filesystem---logical device---physical devices (partitions)

fsck to check filesystem and repair e.g. blocks allocated to more than one inode, missing blocks.
unreferenced files and directories (ie. allocated but no directory entry).
bad link counts, bad inode counts.
physically bad blocks
lost+found directory per filesystem : unreferenced files salvaged by fsck

Booting mounts root filesystem, others can be mounted by /etc/rc.d scripts (mounts your filesystems, cleans up /tmp, starts system daemons (programs that run thruout lifetime of system, eg. printer daemon).

Shutdown: /etc/shutdown script: warn users, kill user and daemon processses, unmount filesystems, sync and halt processor. shutdown -h now
shutdown -r now To reboot.
sync: flush disk buffers to disk (periodically, say once per minute).

NFS multiple filesystems on network transparently sharing Filesystem. Paging from other machines' disks.
NIS (yp) networked machines unique userids, groups, hosts, services

smbfs: mount MS Windows drives
samba: integrate Unix into MS SMB network as file and print server

dosemu: DOS emulation. run DOS programs in Linux
wine: MS Windows emulation (partial).

File descriptors.
Process has three, initially connected to terminal:
0 standard input
1 standard output
2 standard error (for error messages, so if standard output is redirected to file or pipe, error messages still go to terminal).
Commands take input from standard input, send output to standard output, send errors to standard error.
$ echo Hello >f1
$ cat f1
$ cat <f1
$ echo Hello 1>f1
$ cat 0<f1
$ cp f2 f3
cp: cannot access f2     # This error msg was sent to standard error.
$ cp f2 f3 2>errfile  #Redirect standard error to file.  
                    Also, 2>>  append.
$ cat errfile
cp: cannot access f2
$ cp f2 f3 2>/dev/null        # Throw away error msg (not good idea).
$ time sort bigfile >sorted 2>timetosort
time's output
$ (time sort bigfile >sorted) 2>timetosort
2>&1   standard error to standard output.  2 declared a duplicate of 1
$ cp f2 f3 >arf 2>&1                    # stderr to arf too

1>&2 standard output to standard error. Useful for error message in a script whose output has been redirected or sent to a pipe. Prompts in script conventionally to stderr too.
echo "Error..." 1>&2
$ cat chapter2 | tbl | nroff | lp  2>errfile  
# only errors from last cmd redirected
$ (cat chapter2 | tbl | nroff | lp)  2>errfile        
# errors from all cmds in pipeline redirected


Shell variables

shell, as a program, has variables. The variables define the environment of a process. Conventionally, environment variables (those that are exported) are uppercase, others are lowercase.
All are string valued. Usable in scripts.
name=value strings only; name can have letters, digits, _ ; no spaces around =
$ pet=bulldog
$ echo pet
$ echo $pet     # $ is metacharacter to access value of shell variable
$ echo \$pet    # escape the $
Empty a variable by: Null string if not initialized.
Delete a variable by:
unset name

Some variables only needed by shell. Others used by commands (typically to override or augment some default behavior e.g. man grep), so need to be exported to be available to them: environment variables. Passed by value. Most in .bash_profile.
$ set        # display all variables        
                 env shows exported variables
HOME        pathname of home dir
PWD         current directory
DISPLAY     X terminal address
TERM        terminal type
PATH        search path.  list of directories shell will 
                search to find command.  an empty : is current directory.
PS1          prompt: \h host, \u username, \s shell, \W pwd, \d date,
             \t time, \! cmd #, \007 bell,
             \[  \] terminal control sequence within
PS2          secondary prompt
SECONDS     since login
RANDOM      random integer, changes each use
pet             User defined variable

$ echo $HOME
[?Can change PWD=/home/alf but pwd not changed. Changing HOME effects cd]

[Variable attributes, typeset: read-only, integer, display format: lower/upper case, justification]
Shell spawns a child process to run a command in. Shell is a program/command, can be spawned and run as a child process.
$ sh       # create a child shell process. Linux: sh is bash
$                 # are now in that shell

user-defined shell variables are not passed to child shells!
$ x=Hello
$ echo $x
$ sh                 # spawn child shell
$ echo $x            # it doesn't have x

$ exit             # or Ctrl-d  to terminate this shell.  Ctrl-d is end-of-input signal.
$ echo $x       # Now back in original shell

Export to make variables known to all descendants. Pass by value, not by reference.
$ x=Hello
$ export x
$ sh 
$ echo $x
$ x=Bye
$ echo $x
$ sh                   # grandchild 
$ echo $x
Hello       # the originally exported value.   
            ksh exports newer value, i.e. bye
$ ctrl-d             # back to child
$ export x
$ sh                   # (another) grandchild
$ echo $x
$ crtl-d
$ ctrl-d
$ echo $x

unset to delete a variable
$ unset x

Shell scripts (shell files)
sequence of commands in a file (control constructs available too).

Note: all scripts in this text are available in ~wills/bin

Ex. file simp1:
echo Date and time is:
To run script:
1.) spawn a subshell, it reads the script file and executes the commands. sh is actually bash.
$ sh simp1
Date and time is:
Mon 23 Feb 15:32   1996

1a.) run shell, take input from the script file:
$ sh <simp1
Date and time is:
Mon 23 Feb 15:33  1996

Or, 2.) (also spawns subshell to run the commands).
$ chmod +x simp1                # make the file executable
                                  or chmod 700 or 755
$ simp1        # execute the commands in it.   simp1 is now a command!
Date and time is:
Mon 23 Feb  15:35  1996

./simp1   to skip PATH
# shell ignores all chars after this in line. # as start of word
$ #this is a comment
$ echo Hello    #comment here, not an argument of cmd
$ echo Hello#again      #embedded # is not start of comment

 #hi--says Hello
  echo Hello        # an obvious comment,not really needed
  echo Hello#again  # internal # not the start of comment

$ hi

Shell script:
--arguments (on the command line, includes options)
--shell variables
--control constructs: for, if, while, case, until, break, exit

Shell is programmable, is a programming language. Quick to program.
 echo What is your name\?
  read name
  echo Welcome, $name

$ welcome
What is your name?
Welcome, billybob

read name: shell does not interpret. Entire line read into the variable.
joe doe     == joe doe
joe     doe     == joe doe
'joe    doe' == joe    doe
Arguments (positional parameters) to script:
$0 command name (script file name) Ex. cmd that examines its name to determine what action.
$1..$9 first thru ninth argument (can access more than 9 if use braces: ${10})
$* all arguments (but not $0)
$# number of args
 # argtest
  echo Name of script is $0
  echo First arg is $1
  echo There are $# args: $*
$ argtest hi there billybob
Name of script is argtest
First arg is hi
There are 3 args: hi there billybob

$ argtest hi there billybob >jf
$ cat jf
Name of script is argtest
First arg is hi
There are 3 args: hi there billybob
$ argtest hi there billybob |wc
  3   16  78

  # snag--make copy of arg file in backup dir
  cp $1 $HOME/snag/$

$ save coolfile            # cp coolfile /home/billybob/backup/

$ set  a b c          # set assigns values to positional parameters
$ echo $2 $1
b a
 # wh--what's my username
  set `who am i`
  echo $1
$ wh
 #wh--what's my name, and where am I
  set `who am i`
  echo You are $1 on terminal $2

  #cx--give x permission on file args
  chmod +x $*
$ chmod +x cx OR $ sh cx cx
$ cx wh
$ wh
You are billybob on terminal tty03
 #runarg--execute the argument
$ runarg date
Tue 12 Nov 1996 12:32

$ runarg "ps -l"
(output of ps -l)
 #runargs--execute the arguments

$ runargs ls -l
(output of ls -l)
 # gsort--sort lines containing word in file
  grep $1 $2 | sort

$ gsort quark myfile
Want several files as args:
 # gsort--sort lines containing word
  grep $1 $* | sort      # WRONG.  $* includes $1, i.e. the word.

shift "shifts" all args to the left, $2 becomes $1, $3 becomes $2 etc. original $1 gone. $# decremented.
 # gsort--sort lines containing word
  grep $word $* | sort

$ gsort quark fil1 fil2 fil3 fil4
$? exit status. returned by called program to caller, eg. shell
$$ PID of current shell
$! PID of last submitted background process
$ date
Tue 12 Apr 1997  14:23
$ echo $?
0                # 0 is successful "true".  opposite of C.
$ cat f1
cat: cannot open f1
$ echo $?       #[bash always showing 0 ???]
2                              # nonzero is some error (convention). 
       # Different errors have different exit statuses, per  command.
$ cat <f1
f1: cannot open            # from shell. cat not run
$ echo $?

Exit status used in control constructs to indicate true/false value of "condition".
Exit status of previous command can be used to take corrective action (depends on knowing the meaning). Used more in programming.
$ date >>log &                   # stdout of shell, e.g. terminal.    
$ echo $!
$ sh >f         # stdout to file f.  Redirect output of subshell 
            (so output of cmds to f) but get the prompt.  
$ sh >f 2>$1     # no prompt but prompt not in f
$ echo $$
$ ps
PID     TTY         TIME  COMD
4096 ...                               sh
4145 ...                               ps
$$ useful to create unique filenames in script:
 # sf
  sort $1 >$1.$$

The script has a shell spawned to execute the commands in the script.
$ sf f1
$ ls f1*
f1 f1.5749
$ sf f1
$ ls f1*
f1 f1.5749 f1.5753

Ex. One-liner mail script
$ qm billybob Please call John Smith
 #qm--mail a one line message to $1
  #usage: qm username message
  echo $* >temp$$     # args, ie. msg, to file
  mail $recipient <temp$$
  rm temp$$

Conditional execution.
&& do second cmd only if first was successful (0 exit stauts) (i.e. if short-circuit doesn't happen)
|| do second cmd only if first was unsuccessful
  #cgo--compile and go
  cc $1 && a.out
$ cgo prog1.c            # a.out executed only if compile is successful

Ex. within a script. print a file if it contains a specific word:
grep "$word" $filename && lpr $filename

$ mail billybob <letter3 || echo "error mailing"

Not the ksh's && and || within its [[ ]]

Here document.
Ex. <<str take standard input from here, up to next str
script takes input from itself ("here"): script contains its own data.
<<-str allows tab indentation
#medics--lookup a doc
#usage: medics name/specialty
grep -i $1 <<tohere
Joe Lenfant  pediatrics
Jane Roe   obstetrics
Richard Dent  dentistry
John Public  family
Davy Jones  pathology
Al Krebs  oncology
Lou Pasteur epidemiology

$ medics dent
Richard Dent dentistry

More escaping
'  ' escape special meaning of all enclosed chars.   
         strong quote
$ echo 'take this `~\|!@#$%^&*()?.,><-=+" line literally'
take this `~\|!@#$%^&*()?.,><-=+" line literally

" "    escape special meaning of all enclosed chars except:
        weak quote
    $  variable value
    `  command substitution
    \  escape ($  `  "  newline when in pair of double quotes")

Quoting makes one word.
$ echo "take this ~|!@#%^&*()?.,><-=+ line literally"  # no ` \ $
take this ~|!@#%^&*()?.,><-=+ line literally

$ echo "there are `ls|wc -l` files in `pwd`"
there are 37 files in /home/wills/bin

name='John Q. Public'
grep "$name" xfile

$ echo ' "hello" '
$ echo \"hello\"
$ echo " 'hello' "
$ echo ""hello""
$ echo "*"hello"*"

$ fruit=peach
$ echo "the value of fruit is $fruit"
the value of fruit is peach
$ echo "Today is `date`"
Today is Mon 12 Mar  1996

  # wh--what's my name
  set `who am i`
  echo $1
$ wh
$ echo `wh`
$ echo '`wh` is a good student'
`wh` is a good student
$ echo "`wh` is a good student"
billybob is a good student

Quoting to combine several words into one argument.
#repeat--echo the arg
echo $1
$ repeat five
$ repeat five gold rings
$ repeat 'five gold rings'
five gold rings
$ repeat 'five                 shell expects matching end quote
>gold rings'                         > is secondary prompt PS2
five gold rings
$ echo 'five
  #repeat--echo arg
  echo "$1"
$ repeat five
$ repeat 'five gold rings'
five gold rings
$ repeat 'five
>gold rings'
gold rings

$@ all arguments to script. Same as $* except when double quoted:
"$@" is list of double-quoted args
"$*" is double-quoted list of args

Debugging scripts.
Run a shell with options.
-v echo each command before executing it. Trace. set -o verbose
# hi--says Hello
echo Hello
echo Hello again

$ hi
Hello again
$ sh -v hi
# hi--says Hello
echo Hello
echo Hello again
Hello again

set -v turns on tracing. Incorporate tracing within script.
# hi--says Hello
set -v
echo Hello
echo Hello again

$ hi
echo Hello notice set -v not shown
echo Hello again
Hello again

-x show substitutions. set -o xtrace
# wh--what's my name
set `who am i`
echo $1

$ wh
$ sh -x wh
+ who am i
+ set billybob tty03 Mar 3 1996
+ echo billybob
# combined -x and -v
set -o noexec Don't execute script, just syntax check it.

tee command passes input to output but also writes its input to a file. Useful to see what's flowing in a pipe.
... | tee file | ...
... | tee /dev/tty | ...                 # to terminal

script command copies everything that appears on terminal to file typescript. Terminate with end-of-file signal.

Do nothing command
Always successful. Useful as place-holder for 'then' part of If statement, or as always-true condition.

Dot command .
synonym for source command.
Execute commands from a script file in the current shell. Subshell not spawned. Current shell's standard input redirected to the file.
$ . hi
Hello again

Useful for re-executing modified .profile without logging in. Otherwise variable changes made in subshell, not current shell.
$ . .profile

{} run enclosed commands in current shell.
$ {.profile}

for variable in list_of_words

Assign first word in list to variable, do commands. Assign second word in list to variable, do commands. ... Assign last word in list to variable, do commands.
Not like the C/Java for loop!

for, do, done must be first word following separator (newline, ;, &)
 # fortest
  for i in Unix is great
    echo $i

$ fortest

for can be run from command line: shell knows for is not done until done.
$ for i in Friends Romans Countrymen
>echo -e "$i \c" # or $i\\c or $i'\c' OR: -n $i
Friends Romans Countrymen
$ for i in Unix is grrreat;do echo $i; done

Typical use of for: loop over filenames or over all args:
 for i in *

  for i in $*
Ex. print name of files, followed by first line of file:
 #fpeek--first line of file args
  for i in $*
    echo -e "$i : \c"
    head -1 $i

$ fpeek gurufile prog1.cpp
gurufile: Once upon a time there lived a wise guru
prog1.cpp: /* Program prog1.cpp Homework assignment for CMIS 140 */
$ fpeek *

Ex. script to mail a file to several users and record in a log to whom and when file was sent.
$ mlet letter billybob sigmundf
 # mlet--mail file first arg to other args users.  Log file.
  #Usage: mlet filename username(s)
  for person in $*
    mail $person <$letter
    echo $letter sent to $person on `date` >> $HOME/.letlog

 # fortest
  for i in a*      # shell running the script expands the *
    echo $i

$ ls
$ fortest
 # fortest
  for i in 'a*'     #  escape filename expansion.  loops once 
    echo $i         # echo a*, expanded to filenames.  Arg to echo.
$ fortest
a1 a2 a3

# fortest
 for i in 'a*'
    echo "$i"        # echo "a*"
$ fortest
case value in
pattern1) commands1;;
pattern2) commands2;;

Choices are text (string pattern matching). Value compared with choice1, if match then do commands1 then done, else compare value with choice2 and if match do command2 then done,...
Compares done in order specified; first match is done, then quit. Does at most one set of commands (cf. C switch needs break statement).
 # menu--present simple menu, read choice, do it
  echo Choose number to do.
  echo '1 date      2 who'
  echo '3 ls        4 pwd'
  read choice
  case $choice in
  1) date;;
  2) who;;
  3) ls;;
  4) pwd;;
  *) echo Invalid choice;;    # catchall, matches any string
$ menu
Choose number to do.
1 date       2 who
3 ls           4 pwd
4                                              # user's input

Patterns can use shell filename matching operators: * ? [] | (logical or)

Ex. script to delete files named .old, move .c and .f file to a source subdirectory.
  for file in *
    case $file in
     *.old) rm $file
            echo Deleted $file;;
     *.c|*.f) mv $file $HOME/source/$file
              echo moved $file;;

Alternative patterns:

while commands1

Execute the commands1 (usually only one command), if the last is successful ("true") then execute the commands2 of the body, then re-execute the while. Loop until fail, false, exit status non-zero.

Ex. Report the first file in a list that does not contain a certain word.
 # grit-- grep and quit when word not found in file
  # Usage: grit word file(s)
  while grep $word $1 >/dev/null  #throw away any output of grep
  echo First file not containing $word is $1

NB. this grit doesn't work if all files contain the word :(

until commands1

Execute commands1, if last one is not successful then execute commands2 of body then re-execute until.

Ex. Run in the background a process that will report when a certain file has been created in the current directory.
 # isdone--report when file is created
  # Usage: isdone filename
  until find $1 >/dev/null
    sleep 30
  echo ^Gfile $1 is here now

$ isdone myfile &
...(some time later)
$ >myfile
(beep)file myfile is here now

if commands1
fi # semper fi with the if

Execute the commands1, if the last one is successful then execute the commands2.

Ex. compare two files, if they are identical delete the second one.
 # clean--remove a duplicate file
  # Usage: clean file1 file2
  if cmp $1 $2 >/dev/null      # cmp has exit status 0 if files are same contents
    rm $2
    echo $2 was a duplicate and has been deleted

[Oops: clean f1 f1]

if commands1
elif commands3 # elif instead of 'else if'
if date | grep Fri
  echo Bermudas and hawaiians
  echo monkey suit

Ex. see if a word is in either of two files
 #either--finds if word is in either file args
  #Usage: either word file1 file2
  if grep $1 $2 >/dev/null ;then
    echo $1 is in $2
  elif grep $1 $3 >/dev/null ;then
    echo $1 is in $3
    echo $1 is in neither $2 nor $3
 #grits--report all files not containing word
  #usage: grits word files
  for i in $*
    if grep $word $i  >/dev/null
      : #placeholder req'd.  : is the do nothing command
      echo $i does not have $word

! logical not 
if ! grep $word $i >/dev/null
  echo $i does not have $word

A more natural condition:


command: sets exit status
test string1 = string2 # spaces around operator required

Bourne shell only:
$ pet=bulldog
$ test $pet = bulldog
$ echo $?
0 # 0 is "true"
$ test $pet = aardvark
$ echo $?
1 # 1 is "false"
$ test $pet != aardvark
$ echo $?
 # testtest
  echo What is the best OS\?
  read answer
  if test $answer = Unix
    echo Correct!
    echo What\?\?\?

Alternate syntax: [ ] Must be spaces around the brackets. [ is the test command.
if [ $answer = Unix ]

String testing:
 [ -n string  ]     # true if non-zero length
 [ -z string  ]     # true if zero length
 [ string  ]        # true if not null string???
 [ string1 = string2 ]
 [ string1 != string2 ]

[ -n "$1" ] double quote a variable so if it does not exist there will still be an arg for the test
[ "$ans" = yes ]
[ "$ans" = "" ] if ans is empty string, e.g. user entered Enter only.

"" is the explicit null string. if no arguments exist, then $1 is implicit null string.

$ test pet # or [ pet ]
$ echo $?
$ test $pet # assume has value bulldog
$ echo $?
$ test $asdf # does not exist
$ echo $?

File testing:
[ -f file ]      # true if regular file exists
[ -r file ]      # true if file is readable
[ -w file ]     # true if file writable
[ -s file ]     # true if file is not zero length
[ -d file ]     # true if file is a directory

Ex. Copy a file to project dir only if it is readable (bypass cp error msg)
  if [ -r "$1" ]
    cp $1 $HOME/project
    echo $0: $1 not readable

Integer testing:
[ num1 -eq num2 ] # true if the string of digits num1 equals the string of digits num2
Also -ne, -lt, -gt, -ge, -le

Ex. Check for two args:
 if [ $# -ne 2 ]
    echo Need two arguments! 
    exit 1     #quit and set exit status

[ 5 = 5 ]       # true
[ 5 = 05 ]          # false
[ 5 -eq 05 ]    # true
[  10 \< 9 ]  #true!
[ 10 -lt 9 ]    #false

Logical tests:
-a AND
-o OR
[ ! -r myfile ]            # true if myfile is Not readable
[ -r file -a -w file ]     # true if file is readable and is writable
[ -r file -o -w file ]    # true if file is readable or it is writable
[ $answer = Unix -o $answer = UNIX ]
if [ ! -d $dirname ] ;then mkdir $dirname ;fi
if [ ! -r "$1" ] ;then echo "$1" not readable. Bye; exit ;fi

Ex. Cleaner clean. compare two files, if they are identical delete the second one.
 # clean--remove a duplicate file
  # Usage: clean file1 file2
  if [ "$1" != "$2" ]
    if cmp $1 $2 >/dev/null    # cmp has exit status 0 if files are same contents
      rm $2
      echo $2 was a duplicate and has been deleted
      echo $0: same file  

Korn shell [[ ]]
  [[ $1 != $2 ]]       safe if the variables don't exist
  [[ $1 < $2 ]]     instead of -lt
  [[ $1 == 5 && $2 == billybob ]]   instead of -a

Making scripts robust (handle errors).
1. anticipate possible errors of user
2. detect them in script
3. recover from them (possibly terminate script)

User need not know how the script works, so error messages from the commands in the script won't be informative.

Ex. script to copy file into user's home dir, appended by user's name.
 # possess--grab a file for me
  # Usage: possess file
  set `who am i`
  cp $filename $HOME/$filename.$1
$ possess coolfile
# coolfile.billybob in billybob's home directory

Possible errors when running this script:
--wrong number of arguments, i.e. zero
--argument may not be a file, e.g. a directory or not exist
--argument may be file, but not readable by user

Use different exit statuses for different errors with exit command.
 # possess--grab a file for me
  # Usage: possess file
  # check for correct number of args
  case $# in
    1) ;;          #correct number of args
    0) echo Usage: $0 filename  1>&2
       exit 1;;        #terminate script with exit status of 1
    *) echo Warning: only first argument is used  1>&2
  #check for file existence and copy permission
  if test ! -f $1
    echo $0: file $1 not found  1>&2
    exit 2
  elif test ! -r $1
    echo $0: file $1 not readable 1>&2
    exit 3
    set `who am i`
    cp $filename $HOME/$filename.$1

Ex. command to search a file for lines with all words in list (generalized grep)
Method: grep file for first word, then grep the result for second word, then grep that result for third word etc.
  # mgrep--search file for lines with multiple words
  # Usage: mgrep file word(s)
  temp1=/tmp/mgrep1.$$  #temp file with unique name in tmp dir
  for word in $*                  # loop thru word list
    grep $word $file >$temp1
    cp $temp1 $temp2
    file=$temp2                   # search result of last grep
  cat $temp1
  rm $temp1 $temp2              # cleanup the temp files
  exit 0
$ mgrep physicsfile quark lepton boson
(lines of physicsfile that have all 3 words)

Now improve the script.



Generated externally to process and sent to process by kernel.
Asynchronous, like hardware interrupts.
Carry no info (simple kind of interprocess communication, IPC).
Number  Name           How generated and sent to process
  1     hangup HUP      logout                           SIGHUP
  2     interrupt INT   ^C                               SIGINT
  3     quit            ^\     quit and make core dump
  9     KILL            kill -9 pid                      SIGKILL
 10                                                      SIGUSR1
 15     termination     kill pid                         SIGTERM
 19     stop            ^Z   job control                 SIGSTOP

kill -l
lists names and numbers
Normally, all these (except 18) cause process termination. Process can catch signals and act upon them, e.g. ignore it and continue, or take some actions and then continue or quit.

trap 'signal_handler commands' signal_list
From point of execution of trap command to end of script the trap will catch any of the listed signals and instead of terminating will execute the commands and then continue from where the signal occured (unless exit is part of commands).
The trap-handler is invoked for each receipt of the signals. When executing the handler a signal will cause another handler shell to be called. But not every signal of same type will be recognized. No relative priority among signals.

Process might leave some detritus (e.g. temp files) upon termination.
Add to mgrep, near beginning:
trap 'rm $temp1 $temp2; exit 1' 2 3 15

trap ' ' 1
# do nothing if hangup signal received, continue script.

nohup used to be a script that for:
$ nohup cmd
essentially did:
(trap ' ' 1 ; $*) &

Ex. run this and do several ^C's. (Maybe not in sh so can ^Z to suspend the job.)
 # traptest
  trap 'echo Caught signal'  2
  while :
    sleep 10

Kill cmd can send any signal to a process.
$ kill -9 pid
# signal 9 can not be trapped

trap by itself lists signals being trapped.
signal 0 is exit of script.

trap 'echo error at $LINENO' ERR #fake ERR signal of ksh

IPC mechanisms:
--coroutines (parent can output to stdin of child and input from stdout of child)
--pipes (processes must have same parent). From command line. unidirectional flow of data.
--named pipes (processes need not have same parent, nor run at same time)
--queues (multiple putters to and getters from a queue)
--shared memory (multiple readers and writers)
--semaphores (a simple synchronization mechanism, no info transmitted. stop and go). control access to shared resources.
--sockets (end points of two-way communication paths). useful for network protocols; client starts communication from a socket, server listens to a socket to receive the communication.

 # snooze--beeps after $1 seconds
  # Usage: snooze seconds
  sleep $1
  echo ^GWakeup\!        # ASCII 7 (Bell)

$ snooze 40 &
(forty seconds later:)

Periodic awakening:
while always_true e.g. test 1 = 1 or true command
 # snooze--beeps after every $1 seconds
  while true  
    sleep $1
    echo ^GAnother $1 seconds have passed

Could use : do-nothing command which is always successful.
while :

Use a default sleep value if no arg is given:
  # snooze--beeps every $1 or 60 seconds
case $# in
0) sec=60;;
1) sec=$1;;
while :
sleep $sec
echo ^GAnother $sec seconds have passed

Modify so that:
snooze 40 10 30
Check that arguments are numeric-looking.
Do the snooze times in sorted order.
Arithmetic in shell.
expr value operator value the expression is the arguments.
$ expr 5 + 6            # spaces around operator needed
$ expr 9 - 2
$ expr 4 \* 3             # escape *
$ expr 10 / 3            # integer division
$ expr 10 % 3          # mod
$ cows=`expr $legs / 4`

Counting loop in a script:
 read i
  while [ $i -gt 0 ]
    echo $i
    i=`expr $i - 1`    # i = i-1

Korn shell:
let command: (( )) all the C operators

 read i
  while (( i > 0 ))
    echo $i
    (( i=  i-1 ))

$ echo $((i+ i))
Interactive input.
read line from standard input (only)
read a # entire line into a
read a b # first word into a, rest of line into b

 echo Enter your name
  read name
  until [ "$name" ]
    echo No name given, try again
    read name
  echo Welcome $name

Output of a command as the arguments to another command.
xargs [flags] [cmd [(initial args)]]

Ex. search tree of files for string. grep can only do files in current directory.
$ find . -type f -print | xargs grep "word"
#Each filename in output of find is used by xargs as arg to grep.

-i option. standard input to xargs substituted into ()
-p option. prompt ? before each cmd is executed (y to execute)

 #move $1 $2  mv files from dir $1 to dir $2, prompt for each
  ls $1 | xargs -i -p mv $1/() $2/()

Ex. select files of specific type in tree
$ find . -print | xargs file | fgrep " $1" | sed 's/\:\   .*//'
                                                      tab                        tab


Various scripts.

#watchfor--watch for someone to log in
#Usage: watchfor username

case $# in
 0) echo  Usage: $0 user  1>&2; exit 1;;
 1) ;;
 *) echo Warning only first argument used  1>&2;;

until who | grep $1  >/dev/null
  sleep 60

echo ^G$1 logged on now 1>&2
Probably run in background.
Until grep successfully finds user in who listing.

#watchwho--watch who logs in and out
#Usage: watchwho

>$old                #create the old who listing, empty

while :
  who | sort >$new
  diff $old $new
  mv $new $old       # new becomes old for next loop
  sleep 20
done        |   awk '/>/ {$1 = "^Gin:   "; print}
                     /</ {$1 = "^Gout:  "; print}'
Pipe the output of the loop into awk.
diff shows lines that differ between two files. awk program interprets diff's output.

# bundle--group files into distribution package
#  Usage: bundle file(s)

echo '# To unbundle, sh this file'
for i in $*
  echo "echo $i 1>&2"        
  echo "cat >$i <<'End of $i'" # cat the here doc to file
  cat $i                       # here doc 
  echo "End of $i"
$ bundle f1 f2 f3 >f123

# pick--select arguments
# Usage: pick args

for i in $*
  echo -e "$i? \c" >/dev/tty    #prompt user at terminal
  read response
  case $response in
   y*) echo $i;;             # to stdout of pick
   q*) break
done    </dev/tty            #user's input from terminal
$ lp `pick *`
f1? y
myfile? n
new.c? y
prog1.cpp? q
$ pick `cat f1`     #args to pick are contents of f1
$ pick * >f2             #creates list of chosen filenames
$ cp `pick *.txt` Txtfiles/ #copy selected .txt files 
$ lp `pick \`cat f1\``
(contents of f1 as args to pick, thus queries to user)

$ for f in `pick *` ;do mv $f $f.C ;done    
#change name of selected files
#spellchk--check spelling, with user-supplemented file of words
# == spell +file
#Usage: spellchk okwordsfile file

spell $2  |           #output is spell's mispelled words
while read word
  if grep -i "^$word\$" $1 >/dev/null  
    echo $word

#find_links--find the links of arg in subtree
#Usage: find_links file [directory]
if [ $# = 1 ]
  dir='.'        #default is current dir

set -- `ls -l $file`   #get link count

if [ $lnkcnt = 1 ]
  echo $0: no other links 1>&2
  exit 0

set `ls -i $file`    # get inumber

find $dir -inum $inode -print    

#journal--add entries to a journal file
#Usage: journal
date >> $file
echo -e "Enter name of person talked to: \c"
read person
echo $person >> $file
echo >> $file
echo "Enter entry: "
cat >> $file
echo ------------------------------------- >> $file
echo >> $file

#sortmerge--sort and then merge two files
#Usage: sortmerge file1 file2
if [[ $# != 2 ]]
  print "Usage: $0 file1 file2"
  exit 1
trap 'rm f1.$$ f2.$$; exit 2' 1 2 15
#sort the input files into temp files
sort $1 >f1.$$
sort $2 >f2.$$
#set FDs 3 and 4 to the temp files, so can read from them
exec 3<f1.$$
exec 4<f2.$$
#read a line from each sorted file
read -u3 line1
stat1=$?    #true until at eof
read -u4 line2
while [[ $stat1 = 0 && $stat2 = 0 ]]
  if [[ "$line2" > "$line1" ]]
    print $line1
    read -u3 line1
    print $line2
    read -u4 line2
#one of the files is at eof, read from both until at eof
while [[ $stat1 = 0 ]]
  print $line1
  read -u3 line1
while [[ $stat2 = 0 ]]
  print $line2
  read -u4 line2
rm f1.$$ f2.$$

Scripts should be compatible with Unix conventions:
--error-checking, especially arguments
--error messages to standard error
--non-zero exit status for abnormal termination
--options indicated by hyphens, after command name, before other arguments
--cleanup after normal and abnormal termination

Advantages of shell scripts:
--variables, arguments, structured control flow: power of high-level language. operator-rich
--can use Unix commands (primitive statements are themselves entire programs). shell as glue language
--small storage
--easy to change, no compilation; rapid prototyping

Disadvantages: typical of scripting languages
--slow: parse cmd line, search for commands, spawn subprocesses. Interpretted.
--not very arithmetical
--much syntax