What are the shell's control and redirection operators?
I often see tutorials online that connect various commands with different symbols. For example:
command1 | command2 command1 & command2 command1 || command2 command1 && command2
Others seem to be connecting commands to files:
command1 > file1 command1 >> file1
What are these things? What are they called? What do they do? Are there more of them?
These are called shell operators and yes, there are more of them. I will give a brief overview of the most common among the two major classes, control operators and redirection operators, and how they work with respect to the bash shell.
A. Control operators
In the shell command language, a token that performs a control function.
It is one of the following symbols:
& && ( ) ; ;; <newline> | ||
A.1 List terminators
;: Will run one command after another has finished, irrespective of the outcome of the first.
command1 ; command2
command1is run, in the foreground, and once it has finished,
command2will be run.
A newline that isn't in a string literal or after certain keywords is not equivalent to the semicolon operator. A list of
;delimited simple commands is still a list - as in the shell's parser must still continue to read in the simple commands that follow a
;delimited simple command before executing, whereas a newline can delimit an entire command list - or list of lists. The difference is subtle, but complicated: given the shell has no previous imperative for reading in data following a newline, the newline marks a point where the shell can begin to evaluate the simple commands it has already read in, whereas a
;semi-colon does not.
&: This will run a command in the background, allowing you to continue working in the same shell.
command1 & command2
command1is launched in the background and
command2starts running in the foreground immediately, without waiting for
A newline after
A.2 Logical operators
&&: Used to build AND lists, it allows you to run one command only if another exited successfully.
command1 && command2
command2will run after
command1has finished and only if
command1was successful (if its exit code was 0). Both commands are run in the foreground.
This command can also be written
if command1 then command2 else false fi
if command1; then command2; fiif the return status is ignored.
||: Used to build OR lists, it allows you to run one command only if another exited unsuccessfully.
command1 || command2
command2will only run if
command1failed (if it returned an exit status other than 0). Both commands are run in the foreground.
This command can also be written
if command1 then true else command2 fi
or in a shorter way
if ! command1; then command2; fi.
||are left-associative; see Precedence of the shell logical operators &&, || for more information.
!: This is a reserved word which acts as the “not” operator (but must have a delimiter), used to negate the return status of a command — return 0 if the command returns a nonzero status, return 1 if it returns the status 0. Also a logical NOT for the
! command1 [ ! a = a ]
And a true NOT operator inside Arithmetic Expressions:
$ echo $((!0)) $((!23)) 1 0
A.3 Pipe operator
|: The pipe operator, it passes the output of one command as input to another. A command built from the pipe operator is called a pipeline.
command1 | command2
Any output printed by
command1is passed as input to
|&: This is a shorthand for
2>&1 |in bash and zsh. It passes both standard output and standard error of one command as input to another.
command1 |& command2
A.4 Other list punctuation
;;is used solely to mark the end of a case statement. Ksh, bash and zsh also support
;&to fall through to the next case and
;;&(not in ATT ksh) to go on and test subsequent cases.
)are used to group commands and launch them in a subshell.
}also group commands, but do not launch them in a subshell. See this answer for a discussion of the various types of parentheses, brackets and braces in shell syntax.
B. Redirection Operators
In the shell command language, a token that performs a redirection function. It is one of the following symbols:
< > >| << >> <& >& <<- <>
These allow you to control the input and output of your commands. They can appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.
<: Gives input to a command.
command < file.txt
The above will execute
commandon the contents of
<>: same as above, but the file is open in read+write mode instead of read-only:
command <> file.txt
If the file doesn't exist, it will be created.
That operator is rarely used because commands generally only read from their stdin, though it can come handy in a number of specific situations.
>: Directs the output of a command into a file.
command > out.txt
The above will save the output of
out.txt. If the file exists, its contents will be overwritten and if it does not exist it will be created.
command >out.txt 2>error.txt
In the example above,
>will redirect standard output and
2>redirects standard error. Output can also be redirected using
1>but, since this is the default, the
1is usually omitted and it's written simply as
So, to run
file.txtand save its output in
out.txtand any error messages in
error.txtyou would run:
command < file.txt > out.txt 2> error.txt
>|: Does the same as
>, but will overwrite the target, even if the shell has been configured to refuse overwriting (with
set -o noclobber).
command >| out.txt
out.txtexists, the output of
commandwill replace its content. If it does not exist it will be created.
>>: Does the same as
>, except that if the target file exists, the new data are appended.
command >> out.txt
out.txtexists, the output of
commandwill be appended to it, after whatever is already in it. If it does not exist it will be created.
&>>: (non-standard). Redirect both standard error and standard output, replacing or appending, respectively.
command &> out.txt
Both standard error and standard output of
commandwill be saved in
out.txt, overwriting its contents or creating it if it doesn't exist.
command &>> out.txt
As above, except that if
out.txtexists, the output and error of
commandwill be appended to it.
&>variant originates in
bash, while the
>&variant comes from csh (decades earlier). They both conflict with other POSIX shell operators and should not be used in portable
<<: A here document. It is often used to print multi-line strings.
command << WORD Text WORD
commandwill take everything until it finds the next occurrence of
Textin the example above, as input . While
EoFor variations thereof, it can be any alphanumeric (and not only) string you like. When
WORDis quoted, the text in the here document is treated literally and no expansions are performed (on variables for example). If it is unquoted, variables will be expanded. For more details, see the bash manual.
If you want to pipe the output of
command << WORD ... WORDdirectly into another command or commands, you have to put the pipe on the same line as
<< WORD, you can't put it after the terminating WORD or on the line following. For example:
command << WORD | command2 | command3... Text WORD
<<<: Here strings, similar to here documents, but intended for a single line. These exist only in the Unix port or rc (where it originated), zsh, some implementations of ksh, yash and bash.
command <<< WORD
Whatever is given as
WORDis expanded and its value is passed as input to
command. This is often used to pass the content of variables as input to a command. For example:
$ foo="bar" $ sed 's/a/A/' <<< "$foo" bAr # as a short-cut for the standard: $ printf '%s\n' "$foo" | sed 's/a/A/' bAr # or sed 's/a/A/' << EOF $foo EOF
A few other operators (
x<&y) can be used to close or duplicate file descriptors. For details on them, please see the relevant section of your shell's manual (here for instance for bash).
That only covers the most common operators of Bourne-like shells. Some shells have a few additional redirection operators of their own.
Ksh, bash and zsh also have constructs
=(…)(that latter one in
zshonly). These are not redirections, but process substitution.
It would probably be worthwhile noting that not all shells are equal, and specifically highlighting the bash-specific features.
@GregHewgill yeah, I weaseled out of it by saying that I am discussing with respect to `bash`. This is being groomed as a canonical Q&A to close the various "What does this weird thingy do" questions and most of them are from users of bash. I'm hoping someone else will pitch in and answer for non bash shells, but highlighting the bash-specific ones makes a lot of sense. I'll have to check though, I don't know which they are off the top of my head.
`&>`, `>>>`, and `<<<` are all non-posix as is the reference to *not-only* non-alphanum chars in a here-doc's name. This answer also discusses very little about *how* they work - for example, it is almost worse than useless to talk about a *simple command* and a *command* without explaining what these are and how the shell decides.
@mikeserv thanks. They work on bash and zsh though. I don't know what, if anything, is truly bash-specific in that list. I should go through this and add the shells each works in but that would involve finding out first.
And ksh93, yash, mksh, and many others. But they will all likely operate slightly differently in edge-cases between shells - thats why these types of things are iffy.
@mikeserv >>> ? what does that do? where can I find out more about it? I know that you are not talking about >> which is standard.
@hildred - I dunno. Maybe I was just trying to be thorough...? More likely it was a typo. Sorry to disappoint...
The `&&` and `||` uses seem counterintuitive. Doesn't this mean that exit code 0 evaluates to `True` and non-zero evaluates to `False`?
@Arc676 No, they don't evaluate to true or false, that's a completely different context. This just means that an exit value of non-0 indicates a problem (not `false`) and an exit code of 0 indicates success (not `true`). That's always been the way and is quite standard. A non-0 exit code indicates an error in every environment I know of.
It should be noted that these symbols lose their special meaning and become ordinary textual characters when they are quoted (with `'`…`'` or `"`…`"`) or escaped (with ``\``); e.g., ``echo 'Tom & Jerry'`` or `grep \< prog.c`. Bash, specifically, also supports a `$'`…`'` syntax. These quoting methods — `'`…`'` and `"`…`"` (and `$'`…`'`) — differ in ways that are discussed elsewhere.
There's also the `>!` analogue of `>|` to be aware of, in some non-bash shells.
Can this excellent answer be edited to clarify one small aspect of redirects: Is there actually any difference between `cmd1 | cmd2` and `cmd2 < $( cmd1 )`? Why are there two ways, syntactically, to direct stdin from one process to another, and when do shell scripts tend to use one or the other? Are there any commonly-encountered limitations in other usual syntax + operators one can use, or any common confusions, caused by the "<" appearing _after_ the command it's directing into?