Adam's UNIX shells page

Welcome! This page sprang into existence as a result of Alasdair Kergon's CompSoc talk "Shell Secrets", at which talking to Alasdair and a few others gave me the idea that people might find a web page acting as a sort of partial follow-on from the talk both interesting and useful.

I aim to make the page a quick taster of some of the funkier things you can get shells to do, and I also intend to say a bit about zsh, as the talk gave me the impression that many people don't know much about it but are curious and would like to know a bit more.

(It may be worth mentioning that the page will be most accurate to Linux and Solaris users, as that is mainly where my experience lies. Others should find parts of it worthwhile, however.)

Interactive vs. non-interactive

The page will concentrate on the interactive features of shells, i.e. when you are using a shell by interacting with it directly, rather than using it to run scripts or perform other automated tasks. I would personally recommend that on the occasions that you need to do non-interactive shell tasks, you use /bin/sh (which is often symbolically linked to bash or another shell with a sh emulation mode) purely for backwards compatability. sh is adequate for the small range of tasks that suit shell-scripts; if you find that the task is turning out to be more complex than you first thought, you should almost certainly be using Perl, ruby, or python. Please do yourself a favour and do not try to write shell-scripts in csh or tcsh if you have any choice in the matter. See Csh Programming Considered Harmful by Perl/UNIX guru Tom Christiansen if you need convincing of this.

So, with non-interactive shells safely out of the way, let's continue ...

Interactive shells

Basic features

There are some things common to most of the shells which you really should know about. I'm going to assume that you already know about them; if not, make sure to read the manual pages for your shell until you do.

Line editor keypress shortcuts

All standard shells offer at least two keymaps: one based on emacs-style bindings, and one based on vi-style bindings. bash and tcsh default to emacs-style, and zsh defaults according to whether you have $VISUAL or $EDITOR set to vi or emacs.

There are a number of core editing operations that you really should know in order to be able to work efficiently. (Given that emacs-style is more prevalent, I'll list the emacs-style bindings, but if you prefer vi-style, you should ensure you can perform these operations using that keymap instead. My personal opinion is that while vi can be extremely effective for text-editing full text files, the advantages that dual command/insert modality offers are less applicable when editing a single command line.)

Key combination	Effect
Control-A	jump to start of line
Control-E	jump to end of line
Meta-B	move back one word
Meta-F	move forward one word
Control-D	delete character forwards/list completions/log out
Control-K	delete to end of line
Control-U	delete whole line
Meta-Backspace	delete one word to the left
Meta-D	delete one word to the right
Control-T	transpose two characters
Meta-T	transpose two words

N.B. To do Meta- combinations on PC keyboards, try holding down Alt while pressing the other key. If your terminal emulator doesn't support this, instead try pressing (and releasing) ESCAPE before pressing the other key. I find the former more convenient, but the latter is more portable.

For simplicity, from now on I'll abbreviate Control-x as C-x and Meta-x as M-x.)

There are many more which make life easier, and you are strongly advised to read the manual pages to learn as many of them as possible.

Globbing
An example of globbing is typing *.c to mean all files that end in .c, but actually globbing is much more powerful than this. (In fact, under zsh, it is so powerful, it almost makes find redundant.) Search the manual pages of your shell for `glob' to find out more.
Pipelines and redirection
Simply put, pipelines (denoted by the | symbol) allow you to take the output from one command and feed it to another, and redirection (denoted by symbols such as < and >) lets you direct output to a file. Search the manual pages of your shell for `pipeline' and `redirect' to find out more.
Job control
You can set one job going and put it in the background while you get on with other stuff. I'm going to assume that you know about this too; correct me if you don't. Search the manual pages of your shell for background and job to find out more.
Aliases
If you find yourself often typing some commands all the time, you can define an alias which is quicker to type but does the same thing. Search the manual pages of your shell for alias to find out more.

Advanced features

I'm not going to compare them in great detail here, but have a look this FAQ detailing differences in features between the main shells, and in particular the feature matrix. (Please note that this table is rather out of date - for instance, in the zsh column it has N in a few places where it should have Y, and it is missing rows for many of the advanced features that zsh has).

Here are some features I consider absolutely essential for a really good interactive shell:

Interaction with history via short key sequences

I'd say that the two most handy features of zsh and tcsh by a long way are the editor commands history-search-backward and dabbrev-expand, bound by default to the key sequences M-p and M-/ respectively. (See below for use within bash.)
If you type part of a command line and then press M-p, it will search backwards through the history for lines beginning with what you've typed so far. This means that you can very quickly and reliably fetch old lines out of the history. Say that you've just run LaTeX on a file called foo.latex:
```
tcsh$ latex foo.latex
  
```
Then you might run a few other commands (an xdvi maybe), make a few changes and then find that you want to rerun LaTeX on the file. All you need to type is la followed by M-p and the whole latex command magically pops up.
(Sceptics will note that this isn't such a good example because there are other, quicker ways of doing this anyway, e.g. using the emacs LaTeX mode).
M-/ is similar, except that it looks at the current word rather than the current line (a simplistic view of things is that command lines in the shell are built up from words separated with spaces). In other words, you can very quickly and reliably fetch old words out of the history. The power of this is amazing. For example, you might create a new directory:
```
tcsh$ mkdir /usr/local/src/my_project-with.an-awkward_name
  
```
and then maybe type a few other commands. Now you wish to copy all .c files from the current directory into the new directory. Type cp *.c /u and then hit M-/. The shell sees that the current word begins with /u, and looks back through the history for the first word beginning with this, which happens to be the horribly long and difficult-to-type directory name /usr/local/src/my_project-with.an-awkward_name you've just created. Up it pops, saving you much annoying typing. If there was another word in the history also beginning with /u since you created that directory, just keep pressing M-/ unless the right one appears.
I find that these two features alone save me oodles of hassle. bash can be configured to bind M-p and M-n to history-search-backward and history-search-forward respectively; put this in your .bashrc:
```
bind '"\ep":history-search-backward'
bind '"\en":history-search-forward'
  
```
or if you really insist on using bash with vi mode (but see above):
```
bind -m vi-insert '"\ep":history-search-backward'
bind -m vi-insert '"\en":history-search-forward'
  
```
Recent versions (bash 2 or later?) also have M-TAB bound to dynamic-complete-history, which is bash's equivalent of dabbrev-expand. For earlier versions, use:
```
bind '"\e\C-i":dynamic-complete-history'
  
```
(Note that in bash M-/ is already bound to complete-filename.)
zsh has M-p and M-n bound by default. There are two ways of getting it to perform dabbrev-expand-like behaviour. The first is the following trick (thanks to Bart Schaefer for this one):
```
bindkey '\ef' emacs-forward-word
bindkey -s '^]/' '\eb!?\ef?:%\e!'
  
```
This is very simple to set up and should work with even older versions of zsh. However, it does not work well when completing more complex words, and is not configurable.
The best solution is to use a recent version of zsh. (Anything >= 4.0 should do.) These days, it has an unbelievably powerful new completion system (see below), and will provide the _history_complete_word function, which implements the widgets _history-complete-older (bound to M-/) and _history-complete-older (M-,). These are configurable; for instance I have in my .zshrc the following:
```
zstyle ':completion:*:history-words' stop yes
zstyle ':completion:*:history-words' remove-all-dups yes
zstyle ':completion:*:history-words' list false
zstyle ':completion:*:history-words' menu yes
  
```
See the zshcompsys(1) man page for more information on these.
Another very sweet feature is incremental searching, usually bound to C-r (Control R). This is a sort of cross between the other two: press it, and start typing a word which exists somewhere in the history. By the time you've typed enough of it to uniquely identify it, you've plucked the whole of that line out of the history. This is available in bash (C-r by default), tcsh (not bound by default) and zsh (C-r by default).

There are many other cunning shortcuts; read the man pages to find more about them. I'll finish this section with a simple exercise for the reader: find out what M-. and/or M-, are bound to, and see if you can see why I use these many many times every day. If you can't, e-mail me for the answer, since again they are extremely handy!
Fully programmable completion

Completion is when you can press a key (often TAB) and the shell automatically cleverly completes the rest of the word you are typing, based on the context you are typing it in. Most shells only offer basic filename, hostname and command completion. Fully programmable context-sensitive completion is only available in tcsh and zsh by default, and in bash via a separate project. tcsh does it in a fairly messy and inflexible way. zsh has two systems: the older, obsolete one which uses a compctl builtin similar to tcsh's, and the new one which requires virtually no configuration, has out-of-the-box completion for tons of UNIX commands and is extremely sophisticated and powerful, but trivial to use as an end user. The bash programmable completion project is heavily based on this new system, but is much less featureful.
If you haven't seen the full power of programmable completions in action, maybe you might like to give zsh a go. The new programmable completion system (actually it's pretty old now) isn't enabled by default, but it's very easy to set up. You can either use my setup files listed at the end of this page, or see the zshcompsys(1) man page for more information.
If you do try it, type in a few typical UNIX commands, but instead of typing in the arguments/options following a command, press TAB and see what happens. You'll notice that the shell automatically inserts (or gives you the option of inserting) new text based on which bit of the full command you're editing. For example, say you want to extract files from a UNIX tape archive (.tar or .tar.gz file) of the Linux kernel source tree, which you just downloaded to the /tmp directory. You type:
```
/tmp % tar -zxvf l[press TAB]
/tmp % tar -zxvf linux-2.0.38.tar.gz
  
```
If you had chosen -xvf instead, it would complete on files ending in .tar. Say there was also a file called allmail.tar in tmp:
```
/tmp % tar -xvf [press TAB]
/tmp % tar -xvf allmail.tar
  
```
This is where things get really nifty. Say you only want to extract the file zsh-3.1.6-pws-11/Completion/User/_perl_builtin_funcs from zsh-3.1.6-pws-11.tar.gz. No need to type in all of that massive filename. Instead, going through the process step by step:
```
/tmp % tar zxf z[press TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz [press TAB again]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/C[TAB]
ChangeLog       ChangeLog.3.0   Completion/     Config/       
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Com[TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/U[TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_pe[TAB]
_perl_basepods   _perl_builtin_funcs   _perl_modules   _perldoc 
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl[TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl_basepods[TAB again]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl_builtin_funcs
  
```
In fact, you can even configure the shell to get to it quicker using partial directory prefixes in the path (how is left as an exercise for the reader):
```
/tmp % tar zxf z[press TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz [press TAB again]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz z/C/U/_pe[TAB]
_perl_basepods   _perl_builtin_funcs   _perl_modules   _perldoc 
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl[TAB]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl_basepods[TAB again]
/tmp % tar zxf zsh-3.1.6-pws-11.tar.gz zsh-3.1.6-pws-11/Completion/User/_perl_builtin_funcs
  
```
Using completion in this way, you can use the completion system as a way of browsing around directory structures, so you don't need to separate the process of figuring out where something is from the process of retrieving it. Note that this applies equally outside compressed archives, if you were intending to view or edit a file but didn't know exactly where it was, for example. No more typing of endless cd, ls, cd, ls sequences to find a file!
Here are some more nice examples of the zsh extended completion system in action:
- The find command has different completions for each option, e.g. file/directory names are completed for the initial parameter(s), users after the -user option, groups after -group etc. etc. ... zsh is even able to treat the arguments to -exec as a whole new command line!).
- The ssh, ytalk, and finger commands, which complete on your friends' and your usernames and hostnames (automatically appending an @ symbol in between)
- If you have the appropriate trust mechanisms set up within ssh, hitting TAB after scp me@remotehost.com: completes files on the remote user account available for copying back here!
- On rpm-based systems, when you type (say) rpm -qi kernel- and hit TAB, rpm -qa gets called to discover all installed rpms in order to be able to provide completions from all your kernel packages. Not only that, but since rpm -qa can be an expensive operation, the completion function for rpm uses a caching layer so that it only gets called once per database state.
- Hitting TAB after cvs completes a list of all available cvs operations. Hitting TAB after cvs commit or update only completes files changed since the last commit!
- gcc -WTAB completes all warning options.
You certainly get "hardcore hacker" points for fully grokking the internals of the zsh completion system. However, you can rejoice in the knowledge that you won't ever need to in order to enjoy this sophisticated time-saver, since recent versions come with a complete, `out of the box' pre-configured completions system.
Non-broken behaviour

In some cases, tcsh is really broken. Much of the stuff on the above-mentioned Csh Programming Considered Harmful page applies to interactive use of tcsh too.

I used to be a fair fan of tcsh until I discovered zsh (bash 2 didn't exist in those days.)

Why is `zsh` so great?

Have a look at my zsh page.

Changing shell

To try out a new shell, you can just run it from the command line. If you decide to take the plunge and actually want your login shell to change, you'll need to run chsh, or possibly ask your sysadmin, depending on what machine you're using. UNIX shell differences and how to change your shell has more information on this.

Finding out more

As you can see, this page is just a tiny taster of the possibilities which await you. So how do you become a true shell guru?

Read the manual pages from beginning to end and memorise the whole lot. Yep, I'm serious. Only recommended if you are really hardcore. (It's probably worth mentioning that zsh's manual pages are nicely split up into sections.) More practically-minded people should read through looking for things they might find useful, and only bother remembering those bits.
Put your knowledge into practice. It's no good knowing what a funky keypress C-x * is unless you become accustomed to using it when it's handy.
Experiment! Write your own complex completions, or tweak existing ones; write handy little functions, and shell scripts; join the mailing lists; maybe even hack the source ...

Alternatively, if you're a more normal, sane sort of person who hasn't got the time to do all of this, satisfy yourself with learning and getting used to one small feature a week or something. If you choose wisely, the time you spend learning a new feature will be well repaid timewise and from the satisfaction of being able to use your shell a little better.

Here are some gratuitous links in case you are a sucker for more ... (not enough here yet; please mail me with details of other good links).

UNIX shell differences and how to change your shell
The zsh home page
My zsh page, which will hopefully serve as a good starting point for anyone interested in finding out more.

My shell configuration

I've edited and commented my .zshrc (main zsh interactive startup file) for your enjoyment. I strongly suggest that you read through it and edit it before use. Please note that there are other related files.

I've decided not to make available my old tcsh configuration and completion files, as that might encourage people to use tcsh ;-p

Feedback

Well, I hope that was interesting and/or useful. Please feel free to mail me with comments/suggestions/questions if you want.