BASH Frequently Asked Questions

How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

Don't try to use "for". Use a while loop and the read command:

    while read -r line
    do
        echo "$line"
    done < "$file"

The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should always use the -r option with read.

line is a variable name, chosen by you. You can use any valid shell variable name there.

The redirection < "$file" tells the while loop to read from the file whose name is in the variable file. If you would prefer to use a literal pathname instead of a variable, you may do that as well. If your input source is the script's standard input, then you don't need any redirection at all.

If your input source is the contents of a variable/parameter, BASH can iterate over its lines using a "here string":

    while read -r line; do
        echo "$line"
    done <<< "$var"

The same can be done in any Bourne-type shell by using a "here document" (although read -r is POSIX, not Bourne):

    while read -r line; do
        echo "$line"
    done <<EOF
$var
EOF

If avoiding comments starting with # is desired, you can simply skip them inside the loop:

    # Bash
    while read -r line
    do
        [[ $line = \#* ]] && continue
        echo "$line"
    done < "$file"

If you want to operate on individual fields within each line, you may supply additional variables to read:

    # Input file has 3 columns separated by white space.
    while read -r first_name last_name phone; do
      ...
    done < "$file"

If the field delimiters are not whitespace, you can set IFS (internal field separator):

    while IFS=: read -r user pass uid gid gecos home shell; do
      ...
    done < /etc/passwd

For tab-delimited files, use IFS=$'\t'.

You do not necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

    read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'

    # first will contain "Bob", and last will contain "Smith".
    # junk holds everything else.

Some people use the throwaway variable _ as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single read command, if we don't care what goes into it:

    read -r _ _ first middle last _ <<< "$record"

    # We skip the first two fields, then read the next three.
    # Remember, the final _ can absorb any number of fields.
    # It doesn't need to be repeated there.

The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS). If that is not desired, the IFS variable has to be cleared:

    # Exact lines, no trimming
    while IFS= read -r line
    do
        printf '%s\n' "$line"
    done < "$file"

One may also read from a command instead of a regular file:

    some command | while read -r line; do
       other commands
    done

This method is especially useful for processing the output of find with a block of commands:

    find . -type f -print0 | while IFS= read -r -d '' file; do
        mv "$file" "${file// /_}"
    done

This reads one filename at a time from the find command and renames the file, replacing spaces with underscores.

Note the usage of -print0 in the find command, which uses NUL bytes as filename delimiters; and -d '' in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, find and read delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split up those filenames at the newlines and cause the loop body to fail. Additionally it is necessary to set IFS to an empty string, because otherwise read would still strip leading and trailing whitespace. See FAQ #20 for more details.

Using a pipe to send find's output into a while loop places the loop in a SubShell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used after the loop; in that case, see FAQ 24, or use a ProcessSubstitution like:

    while read -r line; do
        other commands
    done < <(some command)

If you want to read lines from a file into an array, see FAQ 5.

1. My text files are broken! They lack their final newlines!

If there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then read will read it but return false, leaving the broken partial line in the read variable(s). You can process this after the loop:

    # Emulate cat
    while IFS= read -r line
    do
        printf '%s\n' "$line"
    done < "$file"
    [ -n "$line" ] && printf %s "$line"

Or:

    # This does not work:
    printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done

    # This does not work either:
    printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"

    # This works:
    printf 'line 1\ntruncated line 2' | (while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line")

For a discussion of why the second example above does not work as expected, see FAQ #24.

2. How to keep other commands from "eating" the input

Some commands greedily eat up all available data on standard input. The examples above do not take precautions against such programs. For example,

    while read -r line
    do
        cat > ignoredfile
        echo "$line"
    done < "$file"

will only print the contents of the first line, with the remaining contents going to "ignoredfile", as cat slurps up all available input.

One workaround is to use a numeric FileDescriptor rather than standard input:

    # Bash
    while read -r -u9 line
    do
        cat > ignoredfile
        echo "$line"
    done 9< "$file"

Or:

    # Bourne
    exec 9< "$file"
    while read line <&9
    do
      ...
    done
    exec 9<&-

This example will wait for the user to type something into the file ignoredfile at each iteration instead of eating up the loop input.

You might need this, for example, with mencoder which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include ssh and ffmpeg. Additional workarounds for this are discussed in FAQ #89.

CategoryShell

How can I store the return value/output of a command in a variable?

Well, that depends on whether you want to store the command's output (either stdout, or stdout + stderr) or its exit status (0 to 255, with 0 typically meaning "success").

If you want to capture the output, you use command substitution:

    output=$(command)      # stdout only; stderr remains uncaptured
    output=$(command 2>&1) # both stdout and stderr will be captured

If you want the exit status, you use the special parameter $? after running the command:

    command
    status=$?

If you want both:

    output=$(command)
    status=$?

The assignment to output has no effect on command's exit status, which is still in $?.

If you don't actually want to store the exit status, but simply want to take an action upon success or failure, just use if:

    if command; then
        echo "it succeeded"
    else
        echo "it failed"
    fi

Or if you want to capture stdout as well as taking action on success/failure, without explicitly storing or checking $?:

    if output=$(command); then
        echo "it succeeded"
    ...

What if you want the exit status of one command from a pipeline? If you want the last command's status, no problem -- it's in $? just like before. If you want some other command's status, use the PIPESTATUS array (BASH only). Say you want the exit status of grep in the following:

    grep foo somelogfile | head -5
    status=${PIPESTATUS[0]}

Bash 3.0 added a pipefail option as well, which can be used if you simply want to take action upon failure of the grep:

    set -o pipefail
    if ! grep foo somelogfile | head -5; then
        echo "uh oh"
    fi

Now, some trickier stuff. Let's say you want only the stderr, but not stdout. Well, then first you have to decide where you do want stdout to go:

    output=$(command 2>&1 >/dev/null)  # Save stderr, discard stdout.
    output=$(command 2>&1 >/dev/tty)   # Save stderr, send stdout to the terminal.
    output=$(command 3>&2 2>&1 1>&3-)  # Save stderr, send stdout to script's stderr.

It's possible, although considerably harder, to let stdout "fall through" to wherever it would've gone if there hadn't been any redirection. This involves "saving" the current value of stdout, so that it can be used inside the command substitution:

    exec 3>&1                    # Save the place that stdout (1) points to.
    output=$(command 2>&1 1>&3)  # Run command.  stderr is captured.
    exec 3>&-                    # Close FD #3.

    # Or this alternative, which captures stderr, letting stdout through:
    { output=$(command 2>&1 1>&3-) ;} 3>&1

In the last example above, note that 1>&3- duplicates FD 3 and stores a copy in FD 1, and then closes FD 3. It could also be written 1>&3 3>&-.

What you cannot do is capture stdout in one variable, and stderr in another, using only FD redirections. You must use a temporary file (or a named pipe) to achieve that one.

Well, you can use a horrible hack like:

   result=$( { stdout=$(cmd) ; } 2>&1; echo "this line is the separator"; echo "$stdout")
   var_out=${result#*this line is the separator$'\n'}
   var_err=${result%$'\n'this line is the separator*}

Obviously, this is not robust, because either the standard output or the standard error of the command could contain whatever separator string you employ.

And if you want the exit code of your cmd (here a modification in the case of if the cmd stdout nothing)

   cmd() { curl -s -v http://www.google.fr; }

   result=$( { stdout=$(cmd); returncode=$?; } 2>&1; echo -n "this is the separator"; echo "$stdout"; exit $returncode)
   returncode=$?

   var_out=${result#*this is the separator}
   var_err=${result%this is the separator*}

Note: the original question read, "How can I store the return value of a command in a variable?" This was, verbatim, an actual question asked in #bash, ambiguity and all.

CategoryShell

How can I find the latest (newest, earliest, oldest) file in a directory?

The tempting solution is to use ls to output sorted filenames and take the first result. As usual, the ls approach cannot be made robust and should never be used in scripts due in part to the possibility of arbitrary characters (including newlines) present in filenames. Therefore, we need some other way to compare file metadata.

The most common requirement is to get the most or least recently modified files in a directory. Bash and all ksh variants can compare modification times (mtime) using the -nt and -ot operators of the conditional expression compound command:

# Bash/ksh
unset -v latest
for file in "$dir"/*; do
  [[ $file -nt $latest ]] && latest=$file
done

Or to find the oldest:

# Bash/ksh
unset -v oldest
for file in "$dir"/*; do
  [[ -z $oldest || $file -ot $oldest ]] && oldest=$file
done

Keep in mind that mtime on directories is that of the most recently added, removed, or renamed file in that directory. Also note that -nt and -ot are not specified by POSIX test, however many shells such as dash include them anyway. No bourne-like shell has analogous operators for comparing by atime or ctime, so one would need external utilities for that; however, it's nearly impossible to either produce output which can be safely parsed, or handle said output in a shell without using nonstandard features on both ends.

If the sorting criteria are different from "oldest or newest file by mtime", then GNU find and GNU sort may be used together to produce a sorted list of filenames + timestamps, delimited by NUL characters. This will of course operate recursively (GNU find's -maxdepth operator can prevent that); Here are a few possibilities, which can be modified as necessary to use atime or ctime, or to sort in reverse order:

# Bash + GNU find + GNU sort (To the precision possible on the given OS, but returns only one result)
IFS= read -r -d '' latest \
  < <(find "$dir" -type f -printf '%T@ %p\0' | sort -znr)
latest=${latest#* }   # remove timestamp + space

# GNU find + Bash w/ arrays (To the nearest 1s, using an undocumented "find -printf" format (%Ts).)
while IFS= read -rd '' 'latest[$(read -rd "" y; echo $y)]'
    do :
done < <(find "$dir" -type f -printf '%p\0%Ts\0')
latest=${latest[-1]}

# GNU stat + Bash /w arrays (non-recursive w/o globstar, to the nearest 1s)
while IFS= read -rd '' 'latest[$(read -rd "" y; echo $y)]'
    do :
done < <(stat '--printf=%n\0%Y\0' "$dir"/*)
latest=${latest[-1]}

One disadvantage to these approaches is that the entire list is sorted, whereas simply iterating through the list to find the minimum or maximum timestamp (assuming we want just one file) would be faster, however, depending on the size of the job the algorithmic disadvantage of sorting may be negligible in comparison to the overhead of using a shell.

# Bash + GNU find
unset -v latest time
while IFS= read -r -d '' line; do
  t=${line%% *} t=${t%.*}   # truncate fractional seconds
  ((t > time)) && { latest=${line#* } time=$t; }
done < <(find "$dir" -type f -printf '%T@ %p\0')

Lastly, here's a more verbose variant for use in a library or .bashrc which can either return a result or assign directly to a variable:

latest() {
    if [[ $FUNCNAME == ${FUNCNAME[1]} ]]; then
        unset -v x latest
        printf ${2:+'-v' "$2"} '%s' "$1"
        return
    fi

    if (($# > 2)); then
        echo $'Usage: latest <glob> <varname>\nError: Takes at most 2 arguments. Glob defaults to *'
        return 1
    fi >&2

    if ! shopt -q nullglob; then
        trap 'shopt -u nullglob; trap - RETURN' RETURN
        shopt -s nullglob
    fi

    local x latest

    for x in ${1-*}; do
        [[ -d $x || $x -nt $latest ]] && latest=$x
    done

    latest "$latest" ${2+"$2"}
}

Readers who are asking this question in order to rotate their log files may wish to look into logrotate(1) instead, if their operating system provides it.

CategoryShell

How can I check whether a directory is empty or not? How do I check for any *.mpg files, or count how many there are?

In Bash, you can do this safely and easily with the nullglob and dotglob options (which change the behaviour of globbing), and an array:

    # Bash
    shopt -s nullglob dotglob
    files=(*)
    (( ${#files[*]} )) || echo directory is empty
    shopt -u nullglob dotglob

Of course, you can use any glob you like instead of *. E.g. *.mpg or /my/music/*.mpg works fine.

Bear in mind that you need read permission on the directory, or it will always appear empty.

Some people dislike nullglob because having unmatched globs vanish altogether confuses programs like ls. Mistyping ls *.zip as ls *.zpi may cause every file to be displayed (for such cases consider setting failglob). Setting nullglob in a SubShell avoids accidentally changing its setting in the rest of the shell, at the price of an extra fork(). If you'd like to avoid having to set and unset shell options, you can pour it all into a SubShell:

    # Bash
    if (shopt -s nullglob dotglob; f=(*); ((! ${#f[@]}))); then
        echo "The current directory is empty."
    fi

The other disadvantage of this approach (besides the extra fork()) is that the array is lost when the subshell exits. If you planned to use those filenames later, then they have to be retrieved all over again.

Both of these examples expand a glob and store the resulting filenames into an array, and then check whether the number of elements in the array is 0. If you actually want to see how many files there are, just print the array's size instead of checking whether it's 0:

    # Bash
    shopt -s nullglob dotglob
    files=(*)
    echo "The current directory contains ${#files[@]} things."

You can also avoid the nullglob if you're OK with putting a non-existing filename in the array should no files match (instead of an empty array):

    # Bash
    shopt -s dotglob
    files=(*)
    if [[ -e ${files[0]} || -L ${files[0]} ]]; then
        echo "The current directory is not empty.  It contains:"
        printf '%s\n' "${files[@]}"
    fi

Without nullglob, if there are no files in the directory, the glob will be added as the only element in the array. Since * is a valid filename, we can't simply check whether the array contains a literal *. So instead, we check whether the thing in the array exists as a file. The -L test is required because -e fails if the first file is a dangling symlink.

If your script needs to run with various non-Bash shell implementations, you can try using an external program like python, perl, or find; or you can try one of these:

    # POSIX
    # Clobbers the positional parameters, so make sure you don't need them.
    set -- *
    if test -e "$1" || test -L "$1"; then
        echo "directory is non-empty"
    fi

At this stage, the positional parameters have been loaded with the contents of the directory, and can be used for processing.

In the Bourne shell, it's even worse, because there is no test -e or test -L:

    # Bourne
    # (Of course, the system must have printf(1).)
    if test "`printf '%s %s %s' .* *`" = '. .. *' && test ! -f '*'
    then
        echo "directory is empty"
    fi

Of course, that fails if * exists as something other than a plain file (such as a directory or FIFO). The absence of a -e test really hurts.

Never try to parse ls output. Even ls -A solutions can break (e.g. on HP-UX, if you are root, ls -A does the exact opposite of what it does if you're not root -- and no, I can't make up something that incredibly stupid).

In fact, one may wish to avoid the direct question altogether. Usually people want to know whether a directory is empty because they want to do something involving the files therein, etc. Look to the larger question. For example, one of these find-based examples may be an appropriate solution:

   # Bourne
   find "$somedir" -type f -exec echo Found unexpected file {} \;
   find "$somedir" -maxdepth 0 -empty -exec echo {} is empty. \;  # GNU/BSD
   find "$somedir" -type d -empty -exec cp /my/configfile {} \;   # GNU/BSD

Most commonly, all that's really needed is something like this:

    # Bourne
    for f in ./*.mpg; do
        test -f "$f" || continue
        mympgviewer "$f"
    done

In other words, the person asking the question may have thought an explicit empty-directory test was needed to avoid an error message like mympgviewer: ./*.mpg: No such file or directory when in fact no such test is required.

Support for a nullglob-like feature is inconsistent. In ksh93 it can be done on a per-pattern basis by prefixing with ~(N)¹:

    # ksh93
    for f in ~(N)*; do
        ....
    done

CategoryShell

5. How can I use array variables?

This answer assumes you have a basic understanding of what arrays are in the first place. If you're new to this kind of programming, you may wish to start with the guide's explanation. This page is more detailed and thorough.

5.1. Intro

BASH and KornShell have one-dimensional arrays indexed by a numerical expression, e.g.:

# Bash
host=(mickey minnie goofy)
n=${#host[*]}
for ((i=0;i<n;i++)); do
    echo "host number $i is ${host[i]}"
done

The indexing always begins with 0, unless you specifically choose otherwise. The awkward expression ${#host[*]} or ${#host[@]} returns the number of elements for the array host. (We'll go into more detail on syntax below.)

Ksh93, Zsh and Bash 4.0 have Associative Arrays as well. These are not available in Bourne, ash, ksh88 or older bash shells and are not specified by POSIX.

POSIX and Bourne shells are not guaranteed to have arrays at all.

BASH and Korn shell arrays are sparse. Elements may be added and deleted out of sequence.

# Bash/ksh
arr[0]=0
arr[1]=1
arr[2]=2
arr[42]="what was the question?"
unset 'arr[2]'
echo "${arr[*]}"
# prints 0 1 what was the question?

You should try to write your code in such a way that it can handle sparse arrays, unless you know in advance that an array will never have holes.

5.2. Loading values into an array

Assigning one element at a time is simple, and portable:

# Bash/ksh
arr[0]=0
arr[42]='the answer'

It's possible to assign multiple values to an array at once, but the syntax differs across shells.

# Bash/ksh93
array=(zero one two three four)

# Korn
set -A array -- zero one two three four

When initializing in this way, the first index will be 0.

You can also initialize an array using a glob (see also NullGlob):

# Bash/ksh93
oggs=(*.ogg)

# Korn
set -A oggs -- *.ogg

or using a substitution of any kind:

# Bash
words=($sentence)
letters=({a..z})    # Bash 3.0 or higher

# Korn
set -A words -- $sentence

When the arrname=(...) syntax is used, any unquoted substitutions inside the parentheses undergo WordSplitting and glob expansion according to the regular shell rules. In the first example above, if any of the words in $sentence contain glob characters, filename expansion may occur.

set -f and set +f may be used to disable and re-enable glob expansion, respectively, so that words like * will not be expanded into filenames. In some scripts, set -f may be in effect already, and therefore running set +f may be undesirable. This is something you must manage properly yourself; there is no easy or elegant way to "store" the glob expansion switch setting and restore it later. (And don't try to say parsing the output of set -o is easy, because it's not.)

5.2.1. Loading lines from a file or stream

In bash 4, the mapfile command (also known as readarray) accomplishes this:

# Bash 4
mapfile -t lines < myfile

# or
mapfile -t lines < <(some command)

See ProcessSubstitution and FAQ #24 for more details on the <() syntax.

mapfile handles blank lines (it inserts them as empty array elements), and it also handles missing final newlines from the input stream. Both those things become problematic when reading data in other ways, as we shall see momentarily.

mapfile does have one serious drawback: it can only handle newlines as line terminators. It can't, for example, handle NUL-delimited files from find -print0.

When mapfile is not available, we have to work very hard to try to duplicate it. There are a great number of ways to almost get it right, but fail in subtle ways.

These examples will duplicate most of mapfile's basic functionality:

# Bash 2.04+, Ksh93
unset lines i
while IFS= read -r; do lines[i++]=$REPLY; done < <(your command)   # or < file
[[ $REPLY ]] && lines[i++]=$REPLY

# Ksh88
unset lines; i=0
while IFS= read -r; do lines[i]=$REPLY; i=$((i+1)); done < file
[ "$REPLY" ] && lines[i]=$REPLY i=$((i+1))

Now let's look at some simpler cases that fail, so you can see why we used such a complicated solution.

Some people might start out like this:

# These examples only work with certain kinds of input files.

# Bash
set -f; IFS=$'\n' lines=($(< myfile)); unset IFS; set +f

# Ksh
set -f; IFS='
'; set -A lines -- $(< myfile); unset IFS; set +f

That's a literal newline (and nothing else) between the single quotes in the Korn shell example.

We use IFS (setting it to a newline) because we want each line of input to become an array element, not each word.

However, relying on IFS WordSplitting causes issues if you have repeated whitespace delimiters, because they will be consolidated. E.g., a file with blank lines will have repeated newline characters. If you wanted the blank lines to be stored as empty array elements, IFS's behavior will backfire on you; the blank lines will disappear. There is no clean workaround for this other than to scrap the whole approach.

# bash
# \v is a vertical tab and rarely/never used, so we can use it to mark empty lines
# additionally bash won't collapse multiple \v .
# now empty lines are preserved in array as empty elements
set -f; IFS=$'\n\v' eval lines='( $(sed -re 's/^$/\v/' myfile) )'; set +f

A second approach would be to read the elements one by one, using a loop. This one does not work (with normal input; ironically, it works with some degenerate inputs):

# Does not work!
unset arr i
while IFS= read -r 'arr[i++]'; do :; done < file

Why doesn't it work? It puts a blank element at the end of the array, because the read -r arr[i++] is executed one extra time after the end of file. However, we'll revisit this approach later.

This one gets us much closer:

# Bash
unset arr i
while read -r; do arr[i++]=$REPLY; done < yourfile

# or
while read -r; do arr[i++]=$REPLY; done < <(your command)

The square brackets create a math context. Inside them, i++ works as a C programmer would expect. (That shortcut works in ksh93, but not in ksh88.)

This approach handles blank lines, but it fails if your file or stream is missing its final newline. So we need to handle that case specially:

# Bash
unset arr i
while read -r; do arr[i++]=$REPLY; done < <(your command)
# Append unterminated data line if there was one.
[[ $REPLY ]] && arr[i++]=$REPLY

This is the "final solution" we gave earlier, handling both blank lines inside the file, and an unterminated final line.

Our second try above (the read -r 'arr[i++]' one) works great if there's an unterminated line (since the array element is populated with the partial data before the exit status of read is checked). Unfortunately, it puts an empty element on the end of the array if the data stream is correctly terminated. So to fix that one, we need to remove the empty element after the loop:

# Bash
unset arr i
while IFS= read -r 'arr[i++]'; do :; done < <(your command)
# Remove trailing empty element, if any.
if [[ ${arr[i-1]} = "" ]]; then unset 'arr[--i]'; fi

This is also a working solution. Whether you prefer to read too many and then have to remove one, or read too few and then have to add one, is a personal choice.

NOTE: it is necessary to quote the 'arr[i++]' passed to read, so that the square brackets aren't interpreted as globs. This is also true for other non-keyword builtins that take a subscripted variable name, such as let and unset.

5.2.2. Reading NUL-delimited streams

If you are trying to deal with records that might have embedded newlines, you will be using an alternative delimiter such as the NUL character ( \0 ) to separate the records. In that case, you'll need to use the -d argument to read as well:

# Bash
unset arr i
while IFS= read -rd '' 'arr[i++]'; do :; done < <(find . -name '*.ugly' -print0)
if [[ ${arr[i-1]} = "" ]]; then unset 'arr[--i]'; fi

# or
while read -rd ''; do arr[i++]=$REPLY; done < <(find . -name '*.ugly' -print0)
[[ $REPLY ]] && arr[i++]=$REPLY

read -d '' tells Bash to keep reading until a NUL byte; normally it reads until a newline. There is no equivalent in Korn shell as far as we're aware.

5.2.3. Appending to an existing array

If you wish to append data to an existing array, there are several approaches. The most flexible is to keep a separate index variable:

# Bash/ksh93
arr[i++]="new item"

If you don't want to keep an index variable, but you happen to know that your array is not sparse, then you can use the highest existing index:

# Bash/ksh
# This will FAIL if the array has holes (is sparse).
arr[${#arr[*]}]="new item"

If you don't know whether your array is sparse or not, but you don't mind re-indexing the entire array (and also being very slow), then you can use:

# Bash
arr=("${arr[@]}" "new item")

# Ksh
set -A arr -- "${arr[@]}" "new item"

If you're in bash 3.1 or higher, then you can use the += operator:

# Bash 3.1
arr+=("new item")

NOTE: the parentheses are required, just as when assigning to an array. (Or you will end up appending to ${arr[0]} which $arr is a synonym for.)

For examples of using arrays to hold complex shell commands, see FAQ #50 and FAQ #40.

5.3. Retrieving values from an array

${#arr[*]} or ${#arr[@]} gives the number of elements in an array:

# Bash
shopt -s nullglob
oggs=(*.ogg)
echo "There are ${#oggs[*]} Ogg files."

* is reported to be quicker than @ when testing on Bash 3. * and @ both seem to work at the same speed when testing on Bash 4.1.

Single elements are retrieved by index:

echo "${foo[0]} - ${bar[j+1]}"

The square brackets are a math context. Arithmetic can be done there, and parameter expansions are done even without $.

Using array elements en masse is one of the key features of shell arrays. In exactly the same way that "$@" is expanded for positional parameters, "${arr[@]}" is expanded to a list of words, one array element per word. For example,

# Korn/Bash
for x in "${arr[@]}"; do
  echo "next element is '$x'"
done

This works even if the elements contain whitespace. You always end up with the same number of words as you have array elements.

If one simply wants to dump the full array, one element per line, this is the simplest approach:

# Bash/ksh
printf "%s\n" "${arr[@]}"

For slightly more complex array-dumping, "${arr[*]}" will cause the elements to be concatenated together, with the first character of IFS (or a space if IFS isn't set) between them. As it happens, "$*" is expanded the same way for positional parameters.

# Bash
arr=(x y z)
IFS=/; echo "${arr[*]}"; unset IFS
# prints x/y/z

Unfortunately, you can't put multiple characters in between array elements using that syntax. You would have to do something like this instead:

# Bash/ksh
arr=(x y z)
tmp=$(printf "%s<=>" "${arr[@]}")
echo "${tmp%<=>}"    # Remove the extra <=> from the end.
# prints x<=>y<=>z

BASH 3.0 added the ability to retrieve the list of index values in an array, rather than just iterating over the elements:

# Bash 3.0 or higher
arr=(0 1 2 3) arr[42]='what was the question?'
unset 'arr[2]'
echo "${!arr[@]}"
# prints 0 1 3 42

Retrieving the indices is extremely important in certain kinds of tasks, such as maintaining parallel arrays with the same indices (a cheap way to mimic having an array of structs in a language with no struct):

# Bash 3.0 or higher
unset file title artist i
for f in ./*.mp3; do
  file[i]=$f
  title[i]=$(mp3info -p %t "$f")
  artist[i++]=$(mp3info -p %a "$f")
done

# Later, iterate over every song.
# This works even if the arrays are sparse, just so long as they all have
# the SAME holes.
for i in "${!file[@]}"; do
  echo "${file[i]} is ${title[i]} by ${artist[i]}"
done

5.3.1. Retrieving with modifications

Bash's Parameter Expansions may be performed on array elements en masse:

# Bash
arr=(abc def ghi jkl)
echo "${arr[@]#?}"          # prints bc ef hi kl
echo "${arr[@]/[aeiou]/}"   # prints bc df gh jkl

Parameter Expansion can also be used to extract elements from an array. Some people call this slicing:

# Bash
echo "${arr[@]:1:3}"        # three elements starting at #1 (second element)
echo "${arr[@]:(-2)}"       # last two elements
echo "${@:(-1)}"            # last positional parameter
echo "${@:(-2):1}"          # second-to-last positional parameter

5.4. Using @ as a pseudo-array

As we see above, the @ array (the array of positional parameters) can be used almost like a regularly named array. This is the only array available for use in POSIX or Bourne shells. It has certain limitations: you cannot individually set or unset single elements, and it cannot be sparse. Nevertheless, it still makes certain POSIX shell tasks possible that would otherwise require external tools:

# POSIX
set -- *.mp3
if [ -e "$1" ]; then
  echo "there are $# MP3 files"
else
  echo "there are 0 MP3 files"
fi

# POSIX
...
# Add an option to our dynamically generated list of options
set -- "$@" -f "$somefile"
...
foocommand "$@"

(Compare to FAQ #50's dynamically generated commands using named arrays.)

6. See Also

• Bash-hackers array documentation

• BashGuide/Arrays

• BashSheet Array reference

• BashFAQ 6 - explaining associative arrays

CategoryShell

7. How can I use variable variables (indirect variables, pointers, references) or associative arrays?

This is a complex page, because it's a complex topic. It's been divided into roughly three parts: associative arrays, evaluating indirect variables, and assigning indirect variables. There are discussions of programming issues and concepts scattered throughout.

7.1. Associative Arrays

We introduce associative arrays first, because in the majority of cases where people are trying to use indirect variable assignments/evaluations, they ought to be using associative arrays instead. For instance, we frequently see people asking how they can have a buch of related variables like IPaddr_hostname1, IPaddr_hostname2 and so on. A more appropriate way to store this data would be in an associative array named IPaddr which is indexed by the hostname.

To map from one string to another, you need arrays indexed by a string instead of a number. These exists in AWK as "associative arrays", in Perl as "hashes", and in Tcl simply as "arrays". They also exist in ksh93, where you'd use them like this:

•  # ksh93
   typeset -A homedir             # Declare ksh93 associative array
   homedir[jim]=/home/jim
   homedir[silvia]=/home/silvia
   homedir[alex]=/home/alex
  
   for user in "${!homedir[@]}"   # Enumerate all indices (user names)
   do
       echo "Home directory of user $user is ${homedir[$user]}"
   done

BASH supports them from version 4 and up:

•  # Bash 4 and up
   declare -A homedir
   homedir[jim]=/home/jim
   # or
   homedir=( [jim]=/home/jim
             [silvia]=/home/silvia
             [alex]=/home/alex )
   ...

Prior to Bash 4 or if you can't use ksh93, your options are limited. Either move to another interpreter (awk, perl, python, ruby, tcl, ...) or re-evaluate your problem to simplify it.

There are certain tasks for which associative arrays are a powerful and completely appropriate tool. There are others for which they are overkill, or simply unsuitable.

Suppose we have several subservient hosts with slightly different configuration, and that we want to ssh to each one and run slightly different commands. One way we could set it up would be to hard-code a bunch of ssh commands in per-hostname functions in a single script and just run them in series or in parallel. (Don't reject this out of hand! Simple is good.) Another way would be to store each group of commands as an element of an associative array keyed by the hostname:

•  source "$conf"
   for host in "${!commands[@]}"; do
       ssh "$host" "${commands[$host]}"
   done
  
   # Where "$conf" is a file like this:
   declare -A commands
   commands=( [host1]="mvn clean install && cd webapp && mvn jetty:run"
              [host2]="..."
   )

This is the kind of approach we'd expect in a high-level language, where we can store hierarchical information in advanced data structures. The difficulty here is that we really want each element of the associative array to be a list or another array of command strings. But the shell simply doesn't permit that kind of data structure.

So, often it pays to step back and think in terms of shells rather than other programming languages. Aren't we just running a script on a remote host? Then why don't we just store the configuration sets as scripts? Then it's simple:

•  # A series of conf files named for the hosts we need to run our commands on:
   for conf in /etc/myapp/*; do
       host=${conf##*/}
       ssh "$host" bash < "$conf"
   done
  
   # /etc/myapp/hostname is just a script:
   mvn clean install &&
   cd webapp &&
   mvn jetty:run

Now we've removed the need for associative arrays, and also the need to maintain a bunch of extremely horrible quoting issues. It is also easy to parallelize using GNU Parallel:

•  parallel ssh {/} bash "<" {} ::: /etc/myapp/*

7.1.1. Associative array hacks in older shells

Before you think of using eval to mimic associative arrays in an older shell (probably by creating a set of variable names like homedir_alex), try to think of a simpler or completely different approach that you could use instead. If this hack still seems to be the best thing to do, consider the following disadvantages:

1. It's really hard to read, to keep track of, and to maintain.

2. The variable names must match the RegularExpression ^[a-zA-Z_][a-zA-Z_0-9]* -- i.e., a variable name cannot contain arbitrary characters but only letters, digits, and underscores. We cannot have a variable's name contain Unix usernames, for instance -- consider a user named hong-hu. A dash '-' cannot be part of a variable name, so the entire attempt to make a variable named homedir_hong-hu is doomed from the start.

3. Quoting is hard to get right. If a content string (not a variable name) can contain whitespace characters and quotes, it's hard to quote it right to preserve it through both shell parsings. And that's just for constants, known at the time you write the program. (Bash's printf %q helps, but nothing analogous is available in POSIX shells.)

4. If the program handles unsanitized user input, it can be VERY dangerous!

Read BashGuide/Arrays or BashFAQ/005 for a more in-depth description and examples of how to use arrays in Bash.

If you need an associative array but your shell doesn't support them, please consider using AWK instead.

7.2. Indirection

7.2.1. Think before using indirection

Putting variable names or any other bash syntax inside parameters is generally a bad idea. It violates the separation between code and data, and as such puts you on a slippery slope toward bugs, security issues, etc. Even when you know you "got it right", because you "know and understand exactly what you're doing", bugs happen to all of us and it pays to respect separation practices to minimize the extent of damage they can cause.

Aside from that, it also makes your code non-obvious and non-transparent.

Normally, in bash scripting, you won't need indirect references at all. Generally, people look at this for a solution when they don't understand or know about Bash Arrays or haven't fully considered other Bash features such as functions.

7.2.2. Evaluating indirect/reference variables

BASH allows you to expand a parameter indirectly -- that is, one variable may contain the name of another variable:

•  # Bash
   realvariable=contents
   ref=realvariable
   echo "${!ref}"   # prints the contents of the real variable

KornShell (ksh93) has a completely different, more powerful syntax -- the nameref command (also known as typeset -n):

•  # ksh93
   realvariable=contents
   nameref ref=realvariable
   echo "$ref"      # prints the contents of the real variable

Unfortunately, for shells other than Bash and ksh93, there is no syntax for evaluating a referenced variable. You would have to use eval, which means you would have to undergo extreme measures to sanitize your data to avoid catastrophe.

It's difficult to imagine a practical use for this that wouldn't be just as easily performed by using an associative array. But people ask it all the time (it is genuinely a frequently asked question).

ksh93's nameref allows us to work with references to arrays, as well as regular scalar variables. For example,

•  # ksh93
   myfunc() {
     nameref ref=$1
     echo "array $1 has ${#ref[*]} elements"
   }
   realarray=(...)
   myfunc realarray

We are not aware of any trick that can duplicate that functionality in POSIX or Bourne shells (short of using eval, which is extremely difficult to do securely). Bash can almost do it -- some indirect array tricks work, and others do not, and we do not know whether the syntax involved will remain stable in future releases. So, consider this a use at your own risk hack.

•  # Bash -- trick #1.  Seems to work in bash 2 and up.
   realarray=(...) ref=realarray; index=2
   tmp="$ref[$index]"
   echo "${!tmp}"            # gives array element [2]
  
   # Bash -- trick #2.  Seems to work in bash 3 and up.
   # Does NOT work in bash 2.05b.
   tmp="$ref[@]"
   printf "<%s> " "${!tmp}"; echo    # Iterate whole array.

It is not possible to retrieve array indices directly using the Bash ${!var} indirect expansion.

7.2.3. Assigning indirect/reference variables

Sometimes you'd like to "point" from one variable to another, for purposes of writing information to a dynamically configurable place. Typically this happens when you're trying to write a "reusable" function, and you want it to put its output in a variable of the caller's choice instead of the function's choice. (Reusability of shell functions is dubious at best, so this is something that should not happen often.)

Assigning a value "through" a reference (or pointer, or indirect variable, or whatever you want to call it -- I'm going to use "ref" from now on) is more widely possible, but the means of doing so are extremely shell-specific.

Before we begin, we must point out that you must control the value of the ref. That is, you should only use a ref whose value you assign within a program, or from trusted input. If an end user can populate the ref variable with arbitrary strings, the result can be unexpected code injection. We'll show an example of this at the end.

In ksh93, we can just use nameref again:

•  # ksh93
   nameref ref=realvariable
   ref="contents"
   # realvariable now contains the string "contents"

In Bash, we can use read and Bash's here string syntax:

•  # Bash
   ref=realvariable
   IFS= read -r $ref <<< "contents"
   # realvariable now contains the string "contents"

However, this only works if there are no newlines in the content. If you need to assign multiline values, keep reading.

A similar trick works for Bash array variables too:

•  # Bash
   aref=realarray
   read -r -a $aref <<< "words go into array elements"
   echo "${realarray[1]}"   # prints "go"

(Again, newlines in the input will break this trick. IFS is used to delimit words, so you may or may not need to set that.)

Another trick is to use Bash's printf -v (only available in recent versions):

•  # Bash 3.1 or higher
   ref=realvariable
   printf -v $ref %s "contents"

The printf -v trick is handy if your contents aren't a constant string, but rather, something dynamically generated. You can use all of printf's formatting capabilities. This trick also permits any string content, including embedded newlines (but not NUL bytes - no force in the universe can put NUL bytes into shell strings usefully). This is the best trick to use if you're in bash 3.1 or higher.

Yet another trick is Korn shell's typeset or Bash's declare. These are roughly equivalent to each other. Both of them cause a variable to become locally scoped to a function, if used inside a function; but if used outside a function, they can operate on global variables.

•  # Korn shell (all versions):
   typeset $ref="contents"
  
   # Bash:
   declare $ref="contents"

Bash 4.2 adds declare -g which can put variables in the global context, even from inside a function.

The advantage of using typeset or declare over eval is that the right hand side of the assignment is not parsed by the shell. If you used eval here, you would have to sanitize/escape the entire right hand side first. This trick also preserves the contents exactly, including newlines, so this is the best trick to use if you're in bash older than 3.1 (or ksh88) and don't need to worry about accidentally changing your variable's scope (i.e., you're not using it inside a function).

However, with bash, you must still be careful about what is on the left-hand side of the assignment. Inside square brackets, expansions are still performed; thus, with a tainted ref, declare can be just as dangerous as eval:

•  # Bash:
   ref='x[$(touch evilfile; echo 0)]'
   ls -l evilfile   # No such file or directory
   declare "$ref=value"
   ls -l evilfile   # It exists now!

This problem also exists with typeset in mksh and pdksh, but apparently not ksh93. This is why the value of ref must be under your control at all times.

If you aren't using Bash or Korn shell, you can do assignments to referenced variables using HereDocument syntax:

•  # Bourne
   ref=realvariable
   read $ref <<EOF
   contents
   EOF

(Alas, read means we're back to only getting at most one line of content. This is the most portable trick, but it's limited to single-line content.)

Remember that, when using a here document, if the sentinel word (EOF in our example) is unquoted, then parameter expansions will be performed inside the body. If the sentinel is quoted, then parameter expansions are not performed. Use whichever is more convenient for your task.

Finally, some people just cannot resist throwing eval into the picture:

•  # Bourne
   ref=myVar
   eval "$ref=\$value"

This expands to the statement that is executed:

•  myVar=$value

The right-hand side is not parsed by the shell, so there is no danger of unwanted side effects. The drawback, here, is that every single shell metacharacter on the right hand side of the = must be escaped carefully. In the example shown here, there was only one. In a more complex situation, there could be dozens.

The good news is that if you can sanitize the right hand side correctly, this trick is fully portable, has no variable scope issues, and allows all content including newlines. The bad news is that if you fail to sanitize the right hand side correctly, you have a massive security hole. Use eval at your own risk.

7.3. See Also

• More advanced indirection on arrays

• BashFAQ/005

• BashGuide/Arrays

• BashSheet Array reference

CategoryShell

8. Is there a function to return the length of a string?

The fastest way, not requiring external programs (but not usable in Bourne shells):

# POSIX
${#varname}

or for Bourne shells:

# Bourne
expr "$varname" : '.*'

(expr prints the number of characters matching the pattern .*, which is the length of the string.)

or:

# Bourne, with GNU expr(1)
expr length "$varname"

(BSD/GNU expr only)

This second version is not specified in POSIX, so is not portable across all platforms. However, if $varname expands to "length", the first version will fail with BSD/GNU expr.

A portable way is:

expr \( "X$varname" : ".*" \) - 1

One may also use awk:

# Bourne
awk -v x="$varname" 'BEGIN {print length(x)}'

Though that one will fail for values of $varname that contain backslash characters, so you may prefer:

# Bourne with POSIX awk
awk  'BEGIN {print length(ARGV[1]);exit}' "$varname"

Similar needs:

# Korn/Bash
${#arrayname[@]}

Returns the number of elements in an array.

# Korn/Bash
${#arrayname[i]}

Returns the length of the array's element i.

CategoryShell

9. How can I recursively search all files for a string?

90% of the time, all you need is one of these:

# Recurse and print matching lines (GNU grep):
grep -r -- "$search" .

# Recurse and print only the filenames (GNU grep):
grep -r -l -- "$search" .

You can use find if your grep lacks a -r option, or if you want to avoid traversing symbolic links:

find . -type f -exec grep -l -- "$search" {} \;

The {} characters will be replaced with the current file name.

This command is slower than it needs to be, because find will call grep with only one file name, resulting in many grep invocations (one per file). Since grep accepts multiple file names on the command line, find can be instructed to call it with several file names at once:

find . -type f -exec grep -l -- "$search" {} +

The trailing '+' character instructs find to call grep with as many file names as possible, saving processes and resulting in faster execution. This example works for POSIX find, e.g. with Solaris, as well as very recent GNU find.

Traditional Unix has a helper program called xargs for the same purpose:

find . -type f | xargs grep -l -- "$search"

However, if your filenames contain spaces or other metacharacters, you'll need to use the BSD/GNU -print0 option:

find . -type f -print0 | xargs -0 grep -l -- "$search"

The -print0 / -0 options ensure that any file name can be processed, even one containing blanks, TAB characters, or newlines.

CategoryShell

10. What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ...

Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) immediately, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the grep command buffers its output, and therefore awk only gets its input in large chunks.

Buffering greatly increases the efficiency of I/O operations, and it's usually done in a way that doesn't visibly affect the user. A simple tail -f from an interactive terminal session works just fine, but when a command is part of a complicated pipeline, the command might not recognize that the final output is needed in (near) real time. Fortunately, there are several techniques available for controlling I/O buffering behavior.

The most important thing to understand about buffering is that it's the writer who's doing it, not the reader.

10.0.1. Eliminate unnecessary commands

In the question, we have the pipeline tail -f logfile | grep 'foo bar' | awk ... (with the actual AWK command being unspecified). There is no problem if we simply run tail -f logfile, because tail -f never buffers its output. Nor is there a problem if we run tail -f logfile | grep 'foo bar' interactively, because grep does not buffer its output if its standard output is a terminal. However, if the output of grep is being piped into something else (such as an AWK command), it starts buffering to improve efficiency.

In this particular example, the grep is actually redundant. We can remove it, and have AWK perform the filtering in addition to whatever else it's doing:

tail -f logfile | awk '/foo bar/ ...'

In other cases, this sort of consolidation may not be possible. But you should always look for the simplest solution first.

10.0.2. Your command may already support unbuffered output

Some programs provide special command line options specifically for this sort of problem:

Each command that writes to a pipe would have to be told to disable buffering, in order for the entire pipeline to run in (near) real time. The last command in the pipeline, if it's writing to a terminal, will not typically need any special consideration.

10.0.3. unbuffer

The expect package has an unbuffer program which effectively tricks other programs into always behaving as if they were being used interactively (which may often disable buffering). Here's a simple example:

tail -f logfile | unbuffer grep 'foo bar' | awk ...

expect and unbuffer are not standard POSIX tools, but they may already be installed on your system.

10.0.4. stdbuf

Recent versions of GNU coreutils (from 7.5 onwards) come with a nice utility called stdbuf, which can be used among other things to "unbuffer" the standard output of a command. Here's the basic usage for our example:

tail -f logfile | stdbuf -oL grep 'foo bar' | awk ...

In the above code, "-oL" makes stdout line buffered; you can even use "-o0" to entirely disable buffering. The man and info pages have all the details.

stdbuf is not a standard POSIX tool, but it may already be installed on your system (if you're using a recent Linux distribution, it will probably be present).

10.0.5. less

If you simply wanted to highlight the search term, rather than filter out non-matching lines, you can use the less program instead of a filtered tail -f:

$ less program.log

• Inside less, start a search with the '/' command (similar to searching in vi). Or start less with the -p pattern option.

• This should highlight any instances of the search term.

• Now put less into "follow" mode, which by default is bound to shift+f.

• You should get an unfiltered tail of the specified file, with the search term highlighted.

"follow" mode is stopped with an interrupt, which is probably control+c on your system. The '/' command accepts regular expressions, so you could do things like highlight the entire line on which a term appears. For details, consult man less.

10.0.6. coproc

If you're using ksh or Bash 4.0+, whatever you're really trying to do with tail -f might benefit from using coproc and fflush() to create a coprocess. Note well that coproc does not itself address buffering issues (in fact it's prone to buffering problems -- hence the reference to fflush). coproc is only mentioned here because whenever someone is trying to continuously monitor and react to a still-growing file (or pipe), they might be trying to do something which would benefit from coprocesses.

10.0.7. Further reading

• http://www.pixelbeat.org/programming/stdio_buffering

CategoryShell

11. How can I recreate a directory hierarchy structure, without the files?

With the cpio program:

 cd "$srcdir" &&
 find . -type d -print | cpio -pdumv "$dstdir"

or with the pax program:

 cd "$srcdir" &&
 find . -type d -print | pax -rwdv  "$dstdir"

or with zsh's special globbing:

 zsh -ec '
 cd -- "$srcdir"
 dirs=(**/*(ND))
 cd -- "$dstdir"
 mkdir -p -- $dirs'

or with GNU tar, and more verbose syntax:

 cd "$srcdir" &&
 find . -type d -print | tar c --files-from - --no-recursion |
   tar x --directory "$dstdir"

This creates a list of directory names with find, non-recursively adds just the directories to an archive, and pipes it to a second tar instance to extract it at the target location. As you can see, tar is the least suited to this task, but people just adore it, so it has to be included here to appease the tar fanboy crowd. (Note: you can't even do this at all with a typical Unix tar. Also note: there is no such thing as "standard tar", as both tar and cpio were intentionally omitted from POSIX in favor of pax.)

All but the zsh solution above will fail if directory names contain newline characters. On many modern BSD/GNU systems, at least, they can be trivially modified to cope with that, by using find -print0 and one of pax -0 or cpio -0 or tar --null (check your system documentation to see which of these commands you have, and which extensions are available).

If you want to create stub files instead of full-sized files, with GNU find(1), the following is likely to be the simplest solution. The find command recreates the regular files using "dummy" files (empty files with the same timestamps):

 cd "$srcdir" &&
 # use one of the above commands first, to make the directories, then:
 find . -type f -exec touch -r {} "$destination"/{} \;

Be aware, though, that according to POSIX, the behaviour of find is unspecified when {} is not standing alone in an argument. Because of this, the following solution is more portable (and probably faster...) than the previous:

 dstdir=whatever; export dstdir
 find . -type f -exec sh -c 'for i; do touch -r "$i" "$dstdir"/"$i"; done' _ {} +

If your find can't handle -exec + then you can use \; instead of + at the end of the command. See UsingFind for explanations.

CategoryShell

12. How can I print the n'th line of a file?

One dirty (but not quick) way is:

    sed -n ${n}p "$file"

But this reads the entire file even if only the third line is desired, which can be avoided by printing line $n using the "p" command, followed by a "q" to exit the script:

    sed -n "$n{p;q;}" "$file"

Another method is to grab the last line from a listing of the first n lines:

   head -n $n $file | tail -n 1

Another approach, using AWK:

   awk "NR==$n{print;exit}" file

If more than one line is needed, it's easy to adapt any of the previous methods:

   x=3 y=4
   sed -n "$x,${y}p;${y}q;" "$file"                # Print lines $x to $y; quit after $y.
   head -n $y "$file" | tail -n $((y - x + 1))     # Same
   head -n $y "$file" | tail -n +$x                # If your tail supports it
   awk "NR>=$x{print} NR==$y{exit}" "$file"        # Same

In Bash 4, a pure-bash solution can be achieved succinctly using the mapfile builtin. More than one line can be read into the array MAPFILE by adjusting the argument to mapfile's -n option:

   mapfile -ts $((n-1)) -n 1 < "$file"
   echo "${MAPFILE[0]}"

mapfile can also be used similarly to head while avoiding buffering issues in the event input is a pipe:

   { mapfile -n $n; head -n 1; } <"$file"

12.1. See Also

• BashFAQ/001

• http://wiki.bash-hackers.org/commands/builtin/mapfile

CategoryShell

13. How do I invoke a shell command from a non-shell application?

You can use the shell's -c option to run the shell with the sole purpose of executing a short bit of script:

    sh -c 'echo "Hi!  This is a short script."'

This is usually pretty useless without a means of passing data to it. The best way to pass bits of data to your shell is to pass them as positional arguments:

    sh -c 'echo "Hi! This short script was run with the arguments: $@"' -- "foo" "bar"

Notice the -- before the actual positional parameters. The first argument you pass to the shell process (that isn't the argument to the -c option) will be placed in $0. Positional parameters start at $1, so we put a little placeholder in $0. This can be anything you like; in the example, we use the generic --.

This technique is used often in shell scripting, when trying to have a non-shell CLI utility execute some bash code, such as with find(1):

    find /foo -name '*.bar' -exec bash -c 'mv "$1" "${1%.bar}.jpg"' -- {} \;

Here, we ask find to run the bash command for every *.bar file it finds, passing it to the bash process as the first positional parameter. The bash process runs the mv command after doing some Parameter Expansion on the first positional parameter in order to rename our file's extension from bar to jpg.

Alternatively, if your non-shell application allows you to set environment variables, you can do that, and then read them using normal variables of the same name.

Similarly, suppose a program (e.g. a file manager) lets you define an external command that an argument will be appended to, but you need that argument somewhere in the middle. In that case:

    #!/bin/sh
    sh -c 'command foo "$1" bar' -- "$@"

CategoryShell

14. How can I concatenate two variables? How do I append a string to a variable?

There is no (explicit) concatenation operator for strings (either literal or variable dereferences) in the shell; you just write them adjacent to each other:

    var=$var1$var2

If the right-hand side contains whitespace characters, it needs to be quoted:

    var="$var1 - $var2"

If you're appending a string that doesn't "look like" part of a variable name, you just smoosh it all together:

    var=$var1/.-

Otherwise, braces or quotes may be used to disambiguate the right-hand side:

    var=${var1}xyzzy
    # Without braces, var1xyzzy would be interpreted as a variable name

    var="$var1"xyzzy
    # Alternative syntax

CommandSubstitution can be used as well. The following line creates a log file name logname containing the current date, resulting in names like e.g. log.2004-07-26:

    logname="log.$(date +%Y-%m-%d)"

There's no difference when the variable name is reused, either. A variable's value (the string it holds) may be reassigned at will:

    string="$string more data here"

Bash 3.1 has a new += operator that you may see from time to time:

    string+=" more data here"     # EXTREMELY non-portable!

It's generally best to use the portable syntax.

CategoryShell

15. How can I redirect the output of multiple commands at once?

Redirecting the standard output of a single command is as easy as:

    date > file

To redirect standard error:

    date 2> file

To redirect both:

    date > file 2>&1

or, a fancier way:

    # Bash only.  Equivalent to date > file 2>&1 but non-portable.
    date &> file

Redirecting an entire loop:

    for i in $list; do
        echo "Now processing $i"
        # more stuff here...
    done > file 2>&1

However, this can become tedious if the output of many programs should be redirected. If all output of a script should go into a file (e.g. a log file), the exec command can be used:

    # redirect both standard output and standard error to "log.txt"
    exec > log.txt 2>&1
    # all output including stderr now goes into "log.txt"

Otherwise, command grouping helps:

    {
        date
        # some other commands
        echo done
    } > messages.log 2>&1

In this example, the output of all commands within the curly braces is redirected to the file messages.log.

More discussion

In-depth: Illustrated Tutorial

CategoryShell

16. How can I run a command on all files with the extension .gz?

Often a command already accepts several files as arguments, e.g.

    zcat -- *.gz

On some systems, you would use gzcat instead of zcat. If neither is available, or if you don't care to play guessing games, just use gzip -dc instead.

The -- prevents a filename beginning with a hyphen from causing unexpected results.

If an explicit loop is desired, or if your command does not accept multiple filename arguments in one invocation, the for loop can be used:

    # Bourne
    for file in ./*.gz
    do
        echo "$file"
        # do something with "$file"
    done

To do it recursively, use find:

    # Bourne
    find . -name '*.gz' -type f -exec do-something {} \;

If you need to process the files inside your shell for some reason, then read the find results in a loop:

    # Bash
    while IFS= read -r file; do
        echo "Now processing $file"
        # do something fancy with "$file"
    done < <(find . -name '*.gz' -print)

This example uses ProcessSubstitution (see also FAQ #24), although a pipe may also be suitable in many cases. However, it does not correctly handle filenames that contain newlines. To handle arbitrary filenames, see FAQ #20.

CategoryShell

17. How can I use a logical AND/OR/NOT in a shell pattern (glob)?

"Globs" are simple patterns that can be used to match filenames or strings. They're generally not very powerful. If you need more power, there are a few options available.

If you want to operate on all the files that match glob A or glob B, just put them both on the same command line:

rm -- *.bak *.old

If you want to use a logical OR in just part of a glob (larger than a single charcter -- for which square-bracketed character classes suffice), in Bash, you can use BraceExpansion:

rm -- *.{bak,old}

If you need something still more general/powerful, in KornShell or BASH you can use extended globs. In Bash, you'll need the extglob option to be set. It can be checked with:

shopt extglob

and set with:

shopt -s extglob

To warm up, we'll move all files starting with foo AND not ending with .d to directory foo_thursday.d:

mv foo!(*.d) foo_thursday.d

A more complex example -- delete all files containing Pink_Floyd AND not containing The_Final_Cut:

rm !(!(*Pink_Floyd*)|*The_Final_Cut*)

By the way: these kind of patterns can be used with the KornShell, too. They don't have to be enabled there, but are the default patterns.

CategoryShell

18. How can I group expressions in an if statement, e.g. if (A AND B) OR C?

The portable (POSIX or Bourne) way is to use multiple test (or [) commands:

    # Bourne
    if test A && test B || test C; then
    ...

The grouping is implicit in this case, because AND (&&) has a higher precedence than OR (||). If we need explicit grouping, then we can use curly braces:

    # Bourne(?)
    if test A && { test B || test C; }; then
    ...

What we should not do is try to use the -a or -o operators of the test command, because the results are undefined.

BASH and KornShell have different, more powerful comparison commands with slightly different (easier) quoting:

• ArithmeticExpression for arithmetic expressions, and

• NewTestCommand for string (and file) expressions.

Examples:

    # Bash/ksh
    if (( (n>0 && n<10) || n == -1 ))
    then echo "0 < $n < 10, or n==-1"
    fi

or

    # Bash/ksh
    if [[ ( -f $localconfig && -f $globalconfig ) || -n $noconfig ]]
    then echo "configuration ok (or not used)"
    fi

Note that the distinction between numeric and string comparisons is strict. Consider the following example:

    n=3
    if [[ $n>0 && $n<10 ]]
    then echo "$n is between 0 and 10"
    else echo "ERROR: invalid number: $n"
    fi

The output will be "ERROR: ....", because in a string comparision "3" is bigger than "10", because "3" already comes after "1", and the next character "0" is not considered. Changing the square brackets to double parentheses (( makes the example work as expected.

CategoryShell

19. How can I use numbers with leading zeros in a loop, e.g. 01, 02?

As always, there are different ways to solve the problem, each with its own advantages and disadvantages.

Bash version 4 allows zero-padding and ranges in its BraceExpansion:

    # Bash 4
    echo {01..10}

    for i in {01..10}; do ...

All of the other solutions on this page will assume Bash earlier than 4.0, or a non-Bash shell.

If there are not many numbers, BraceExpansion can be used:

    # Bash
    for i in 0{1,2,3,4,5,6,7,8,9} 10
    do
        echo $i
    done

In Bash 3, you can use ranges inside brace expansion (but not zero-padding). Thus, the same thing can be accomplished more concisely like this:

    # Bash 3
    for i in 0{1..9} 10
    do
        echo $i
    done

Another example, for output of 0000 to 0034:

    # Bash 3
    for i in {000{0..9},00{10..34}}
    do
        echo $i
    done

    # using the outer brace instead of just adding them one next to the other
    # allows to use the expansion, for instance, like this:
    wget 'http://foo.com/adir/thepages'{000{0..9},00{10..34}}'.html'

Some may prefer the following quick & dirty solution (producing "001" through "015"):

    # Bash 3
    for i in {1000..1015}
    do
      echo "${i:1}"    # or "${i#1}"
    done

This gets tedious for large sequences, but there are other ways, too. If you have the printf command (which is a Bash builtin, and is also POSIX standard), it can be used to format a number:

    # Bash
    for ((i=1; i<=10; i++))
    do
        printf "%02d " "$i"
    done

Also, since printf will implicitly loop if given more arguments than format specifiers, you can simplify this enormously:

   # Bash 3
   printf "%03d\n" {1..300}

If you don't know in advance what the starting and ending values are:

   # Bash 3
   # start and end are variables containing integers
   eval printf '"%03d\n"' {$start..$end}

The eval is needed here because you cannot have variables in a brace expansion -- only constants. The extra quotes are required by the eval so that our \n isn't changed to an n. Given how messy that eval solution is, please give serious thought to using the for loop instead.

The KornShell has the typeset command to specify the number of leading zeros:

    # Korn
    $ typeset -Z3 i=4
    $ echo $i
    004

If the command seq(1) is available (it's part of GNU sh-utils/coreutils), you can use it as follows:

    seq -w 1 10

or, for arbitrary numbers of leading zeros (here: 3):

    seq -f "%03g" 1 10

Combining printf with seq(1), you can do things like this:

   # POSIX shell, GNU utilities
   printf "%03d\n" $(seq 300)

(That may be helpful if you are not using Bash, but you have seq(1), and your version of seq(1) lacks printf-style format specifiers. That's a pretty odd set of restrictions, but I suppose it's theoretically possible. Since seq is a nonstandard external tool, it's good to keep your options open.)

Be warned however that using seq might be considered bad style; it's even mentioned in Don't Ever Do These.

Some BSD-derived systems have jot(1) instead of seq(1). In accordance with the glorious tradition of Unix, it has a completely incompatible syntax:

   # POSIX shell, OpenBSD et al.
   printf "%02d\n" $(jot 10 1)

   # Bourne shell, OpenBSD (at least)
   jot -w %02d 10 1

Finally, the following example works with any BourneShell derived shell (which also has expr and sed) to zero-pad each line to three bytes:

   # Bourne
   i=0
   while test $i -le 10
   do
       echo "00$i"
       i=`expr $i + 1`
   done |
       sed 's/.*\(...\)$/\1/g'

In this example, the number of '.' inside the parentheses in the sed command determines how many total bytes from the echo command (at the end of each line) will be kept and printed.

But if you're going to rely on an external Unix command, you might as well just do the whole thing in awk in the first place:

   # Bourne
   # count variable contains an integer
   awk -v count="$count" 'BEGIN {for (i=1;i<=count;i++) {printf("%03d\n",i)} }'

   # Bourne, with Solaris's decrepit and useless awk:
   awk "BEGIN {for (i=1;i<=$count;i++) {printf(\"%03d\\n\",i)} }"

Now, since the number one reason this question is asked is for downloading images in bulk, you can use the examples above with xargs(1) and wget(1) to fetch files:

   almost any example above | xargs -i% wget $LOCATION/%

The xargs -i% will read a line of input at a time, and replace the % at the end of the command with the input.

Or, a simpler example using a for loop:

   # Bash 3
   for i in {1..100}; do
      wget "$prefix$(printf %03d $i).jpg"
      sleep 5
   done

Or, avoiding the subshells (requires bash 3.1):

   # Bash 3.1
   for i in {1..100}; do
      printf -v n %03d $i
      wget "$prefix$n.jpg"
      sleep 5
   done

CategoryShell

20. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?

Some Unix systems provide the split utility for this purpose:

    split --lines 10 --numeric-suffixes input.txt output-

For more flexibility you can use sed. The sed command can print e.g. the line number range 1-10:

    sed -n -e '1,10p' -e '10q'

This stops sed from printing each line (-n). Instead it only processes the lines in the range 1-10 ("1,10"), and prints them ("p"). The command will quit after reading line 10 ("10q").

We can now use this to print an arbitrary range of a file (specified by line number):

# POSIX shell
file=/etc/passwd
range=10
cur=1
last=$(wc -l < "$file") # count number of lines
chunk=1
while [ $cur -lt $last ]
do
    endofchunk=$(($cur + $range - 1))
    sed -n -e "$cur,${endofchunk}p" -e "${endofchunk}q" "$file" > chunk.$(printf %04d $chunk)
    chunk=$(($chunk + 1))
    cur=$(($cur + $range))
done

The previous example uses POSIX arithmetic, which older Bourne shells do not have. In that case the following example should be used instead:

# legacy Bourne shell; assume no printf either
file=/etc/passwd
range=10
cur=1
last=`wc -l < "$file"` # count number of lines
chunk=1
while test $cur -lt $last
do
    endofchunk=`expr $cur + $range - 1`
    sed -n -e "$cur,${endofchunk}p" -e "${endofchunk}q" "$file" > chunk.$chunk
    chunk=`expr $chunk + 1`
    cur=`expr $cur + $range`
done

Awk can also be used to produce a more or less equivalent result:

 awk -v range=10 '{print > FILENAME "." (int((NR -1)/ range)+1)}' file

CategoryShell

21. How can I find and deal with file names containing newlines, spaces or both?

First and foremost, to understand why you're having trouble, read Arguments to get a grasp on how the shell understands the statements you give it. It is vital that you grasp this matter well if you're going to be doing anything with the shell.

The preferred method to deal with arbitrary filenames is still to use find(1):

find ... -exec command {} \;

or, if you need to handle filenames en masse:

find ... -exec command {} +

xargs is rarely ever more useful than the above, but if you really insist, remember to use -0:

# Requires GNU/BSD find and xargs
find ... -print0 | xargs -0 command

# Never use xargs without -0 or similar extensions!

Use one of these unless you really can't.

Another way to deal with files with spaces in their names is to use the shell's filename expansion (globbing). This has the disadvantage of not working recursively (except with zsh's extensions or bash 4's globstar), but if you just need to process all the files in a single directory, it works fantastically well.

For example, this code renames all the *.mp3 files in the current directory to use underscores in their names instead of spaces:

# Bash/ksh
for file in ./*\ *.mp3; do
  mv "$file" "${file// /_}"
done

For more examples of renaming files, see FAQ #30.

Remember, you need to quote all your Parameter Expansions using double quotes. If you don't, the expansion will undergo WordSplitting (see also argument splitting and BashPitfalls). Also, always prefix globs with "./"; otherwise, if there's a file with "-" as the first character, the expansions might be misinterpreted as options.

Another way to handle filenames recursively involves using the -print0 option of find (a GNU/BSD extension), together with bash's -d option for read:

# Bash
unset a i
while IFS= read -r -d $'\0' file; do
  a[i++]="$file"        # or however you want to process each file
done < <(find /tmp -type f -print0)

The preceding example reads all the files under /tmp (recursively) into an array, even if they have newlines or other whitespace in their names, by forcing read to use the NUL byte (\0) as its line delimiter. Since NUL is not a valid byte in Unix filenames, this is the safest approach besides using find -exec. IFS= is required to avoid trimming leading/trailing whitespace, and -r is needed to avoid backslash processing. In fact, $'\0' is equivalent to '' so we could also write it like this:

# Bash
unset a i
while IFS= read -r -d '' file; do
  a[i++]="$file"
done < <(find /tmp -type f -print0)

So, why doesn't this work?

# DOES NOT WORK
unset a i
find /tmp -type f -print0 | while IFS= read -r -d '' file; do
  a[i++]="$file"
done

Because of the pipeline, the entire while loop is executed in a SubShell and therefore the array assignments will be lost after the loop terminates. (For more details about this, see FAQ #24.)

CategoryShell

How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?

There are a number of tools available for this. Which one to use depends on a lot of factors, the biggest of which is of course what we're editing.

First, we'll start with how to replace strings in:

1. Variables

If it's a variable, this can (and should) be done very simply with parameter expansion. Forking an external tool for string manipulation is extremely slow and unnecessary.

var='some string'; search=some; rep=another

# Bash
var=${var//"$search"/$rep}


# POSIX function

# usage: string_rep SEARCH REPL STRING
# replaces all instances of SEARCH with REPL in STRING
string_rep() {
  # initialize vars
  in=$3
  unset out

  # SEARCH must not be empty
  [[ $1 ]] || return

  while true; do
    # break loop if SEARCH is no longer in "$in"
    case "$in" in
      *"$1"*) : ;;
      *) break;;
    esac

    # append everything in "$in", up to the first instance of SEARCH, and REP, to "$out"
    out=$out${in%%"$1"*}$2
    # remove everything up to and including the first instance of SEARCH from "$in"
    in=${in#*"$1"}
  done

  # append whatever is left in "$in" after the last instance of SEARCH to out, and print
  printf '%s%s\n' "$out" "$in"
}

var=$(string_rep "$search" "$rep" "$var")

# Note: POSIX does not have a way to localize variables. Most shells (even dash and 
# busybox), however, do. Feel free to localize the variables if your shell supports
# it. Even if it does not, if you call the function with var=$(string_rep ...), the
# function will be run in a subshell and any assignments it makes will not persist.

In the bash example, the quotes around "$search" prevent the contents of the variable to be treated as a shell pattern (also called a "glob"). Of course, if pattern matching is intended, do not include the quotes. If "$rep" were quoted, however, the quotes would be treated as literal.

Parameter expansions like this are discussed in more detail in Faq #100.

2. Streams

If it's a file or a stream, things get a little bit trickier. The standard tools available for this are sed or AWK (for streams), and ed (for files).

Of course, you could do it in bash itself, by combining the previous method with Faq #1:

search=foo; rep=bar

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < <(some_command)

some_command | while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done

If you want to do more processing than just a simple search/replace, this may be the best option. Note that the last example runs the loop in a subshell. See Faq #24 for more information on that.

Another option would, of course, be sed:

# replaces all instances of "search" with "replace" in the output of "some_command"
some_command | sed 's/search/replace/g'

sed uses regular expressions. Unlike the bash, "search" and "replace" would have to be rigorously escaped in order to treat the values as literal strings. This is very impractical, and attempting to do so will make your code extremely prone to bugs. Embedding shell variables in sed is never a good idea.

You may notice, however, that the bash loop above is very slow for large data sets. So how do we find something faster, that can replace literal strings? Well, you could use AWK. The following function replaces all instances of STR with REP, reading from stdin and writing to stdout.

# usage: gsub_literal STR REP
# replaces all instances of STR with REP. reads from stdin and writes to stdout.
gsub_literal() {
  # STR cannot be empty
  [[ $1 ]] || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

some_command | gsub_literal "$search" "$rep"


# condensed as a one-liner:
some_command | awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'

3. Files

Actually editing files gets even trickier. The only tool listed that actually edits a file is ed. The other methods could be used, but to do so involves a temp file and mv (or POSIX extensions).

ed is the standard UNIX command-based editor. Here are some commonly-used syntaxes for replacing the string olddomain.com by the string newdomain.com in a file named file. All four commands do the same thing, with varying degrees of portability and efficiency:

# Bash
ed -s file <<< $'g/olddomain\\.com/s//newdomain.com/g\nw\nq'

# Bourne (with printf)
printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ed -s file

printf 'g/olddomain\\.com/s//newdomain.com/g\nw\nq' | ed -s file

# Bourne (without printf)
ed -s file <<!
g/olddomain\\.com/s//newdomain.com/g
w
q
!

To replace a string in all files of the current directory:

for file in ./*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done

To do this recursively, the easy way would be to enable globstar in bash 4 (shopt -s globstar, a good idea to put this in your ~/.bashrc) and use:

for file in ./**/*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done

If you don't have bash 4, you can use find. Unfortunately, it's a bit tedious to feed ed stdin for each file hit:

find . -type f -exec bash -c 'printf "%s\n" "g/old/s//new/g" w q | ed -s "$1"' _ {} \;

sed is a Stream EDitor, not a file editor. Nevertheless, people everywhere tend to abuse it for trying to edit files. It doesn't edit files. GNU sed (and some BSD seds) have a -i option that makes a copy and replaces the original file with the copy. An expensive operation, but if you enjoy unportable code, I/O overhead and bad side effects (such as destroying symlinks), this would be an option:

sed -i    's/old/new/g' ./*  # GNU
sed -i '' 's/old/new/g' ./*  # BSD

# POSIX sed, uses a temp file and mv:

# remove all temp files on exit, in case sed fails and they weren't moved
trap 'rm -f "${temps[@]}"' EXIT

temps=()
for file in ./*; do
  if [[ -f $file ]]; then
    tmp=$(mktemp) || exit
    temps+=("$tmp")

    sed 's/old/new/g' "$file" > "$tmp" &&
    mv "$tmp" "$file"
  fi
done

Those of you who have perl 5 can accomplish the same thing using this code:

perl -pi -e 's/old/new/g' ./*

Moreover, perl can be used to pass variables into both search and replace strings with no unquoting or potential for conflict with sigil characters:

in="input (/string" out="output string" perl -pi -e $'$quoted_in=quotemeta($ENV{\'in\'}); s/$quoted_in/$ENV{\'out\'}/g' ./*

Recursively using find:

find . -type f -exec perl -pi -e 's/old/new/g' {} \;   # if your find doesn't have + yet
find . -type f -exec perl -pi -e 's/old/new/g' {} +    # if it does

If you want to delete lines instead of making substitutions:

# Deletes any line containing the perl regex foo
perl -ni -e 'print unless /foo/' ./*

To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...:

find . -type f -exec perl -i.bak -pne \
    's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' {} \;

All of the tools listed above use regular expressions, which means they have the same issue as the sed code earlier; trying to embed shell variables in them is a terrible idea, and treating an arbitrary value as a literal string is painful at best. This brings us back to our while read loop, or the awk function above.

The while read loop:

# overwrite a single file
tmp=$(mktemp) || exit
trap 'rm -f "$tmp"' EXIT

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file" > "$tmp" && mv "$tmp" "$file"

Replaces all files in a directory:

trap 'rm -f "${temps[@]}"' EXIT

temps=()
for f in ./*; do
  if [[ -f $f ]]; then
    tmp=$(mktemp) || exit
    temps+=("$tmp")

    while IFS= read -r line; do
      printf '%s\n' "${line//"$search"/$rep}"
    done < "$f" > "$tmp" &&
    mv "$tmp" "$f"
  fi
done

The above glob could be changed to './**/*' in order to use globstar (mentioned above) to be recursive, or of course we could use find:

# this example uses GNU find's -print0. Using POSIX find -exec is left as an exercise to the reader
trap 'rm -f "${temps[@]}"' EXIT

temps=()
while IFS= read -rd '' f <&3; do
  tmp=$(mktemp) || exit
  temps+=("$tmp")

  while IFS= read -r line; do
    printf '%s\n' "${line//"$search"/$rep}"
  done < "$f" > "$tmp" &&
  mv "$tmp" "$f"
done 3< <(find . -type f -print0)

And of course, we can adapt the AWK function above. The following function replaces all instances of STR with REP in FILE, actually overwriting FILE:

# usage: gsub_literal_f STR REP FILE
# replaces all instances of STR with REP in FILE
gsub_literal_f() {
  local tmp
  # make sure FILE exists, is a regular file, and is readable and writable
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  # STR cannot be empty
  [[ $1 ]] || return

  tmp=$(mktemp) || return
  trap 'rm -f "$tmp"' RETURN

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  ' "$3" > "$tmp" && mv "$tmp" "$3"
}

This function, of course, could be called on all of the files in a dir, or recursively.

Notes:

For more information on sed or awk, you can visit the ##sed and #awk channels on freenode, respectively.

mktemp(1), used in many of the examples above, is not completely portable. While it will work on most systems, more information on safely creating temp files can be found in Faq #62.

CategoryShell

How can I calculate with floating point numbers instead of just integers?

BASH's builtin arithmetic uses integers only:

$ echo $((10/3))
3

For most operations involving floating-point numbers, an external program must be used, e.g. bc, AWK or dc:

$ echo "scale=3; 10/3" | bc
3.333

The "scale=3" command notifies bc that three digits of precision after the decimal point are required.

Same example with dc (reversed polish calculator, lighter than bc):

$ echo "3 k 10 3 / p" | dc
3.333

k sets the precision to 3, and p prints the value of the top of the stack with a newline. The stack is not altered, though.

If you are trying to compare floating point numbers (less-than or greater-than), and you have GNU bc, you can do this:

# Bash
if (( $(bc <<< "1.4 < 2.5") )); then
  echo "1.4 is less than 2.5."
fi

However, x < y is not supported by all versions of bc:

# This would work with some versions, but not HP-UX 10.20.
imadev:~$ bc <<< '1 < 2'
syntax error on line 1,

If you want to be portable, you need something more subtle:

# POSIX
case $(echo "1.4 - 2.5" | bc) in
  -*) echo "1.4 is less than 2.5";;
esac

This example subtracts 2.5 from 1.4, and checks the sign of the result. If it is negative, the first number is less than the second. We aren't actually treating bc's output as a number; we're treating it as a string, and only looking at the first character.

Legacy (Bourne) version:

# Bourne
case "`echo "1.4 - 2.5" | bc`" in
  -*) echo "1.4 is less than 2.5";;
esac

AWK can be used for calculations, too:

$ awk 'BEGIN {printf "%.3f\n", 10 / 3}'
3.333

There is a subtle but important difference between the bc and the awk solution here: bc reads commands and expressions from standard input. awk on the other hand evaluates the expression as part of the program. Expressions on standard input are not evaluated, i.e. echo 10/3 | awk '{print $0}' will print 10/3 instead of the evaluated result of the expression.

Newer versions of zsh and the KornShell have built-in floating point arithmetic, together with mathematical functions like sin() or cos(). So many of these calculations can be done natively in ksh:

# ksh93
$ echo $((3.00000000000/7))
0.428571428571428571

Comparing two floating-point numbers for equality is actually an unwise thing to do; two calculations that should give the same result on paper may give ever-so-slightly-different floating-point numeric results due to rounding/truncation issues. If you wish to determine whether two floating-point numbers are "the same", you may either:

• Round them both to a desired level of precision, and then compare the rounded results for equality; or

• Subtract one from the other and compare the absolute value of the difference against an epsilon value of your choice.

One of the very few things that Bash actually can do with floating-point numbers is round them, using printf:

# Bash 3.1
# See if a and b are close to each other.
# Round each one to two decimal places and compare results as strings.
a=3.002 b=2.998
printf -v a1 %.2f $a
printf -v b1 %.2f $b
if [[ $a1 = "$b1" ]]; then echo "a and b are the same, roughly"; fi

Caveat: Many problems that look like floating point arithmetic can in fact be solved using integers only, and thus do not require these tools -- e.g., problems dealing with rational numbers. For example, to check whether two numbers x and y are in a ratio of 4:3 or 16:9 you may use something along these lines:

# Bash
# Variables x and y are integers
if (( $x*9-$y*16==0 )) ; then
   echo "16:9."
elif (( $x*3-$y*4==0 )) ; then
   echo "4:3."
else
   echo "Neither 16:9 nor 4:3."
fi

A more elaborate test could tell if the ratio is closest to 4:3 or 16:9 without using floating point arithmetic. Note that this very simple example that apparently involves floating point numbers and division is solved with integers and no division. If possible, it's usually more efficient to convert your problem to integer arithmetic than to use floating point arithmetic.

CategoryShell

I want to launch an interactive shell that has special aliases and functions, not the ones in the user's ~/.bashrc.

Just specify a different start-up file:

bash --rcfile /my/custom/bashrc

Variant question: I have a script that sets up an environment, and I want to give the user control at the end of it.

Put exec bash at the end of it to launch an interactive shell. This shell will inherit the environment (which does not include aliases, but that's OK, because aliases suck). Of course, you must also make sure that your script runs in a terminal -- otherwise, you must create one, for example, by using exec xterm -e bash.

CategoryShell

I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?

In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:

    # Works only in ksh88/ksh93, or bash 4.2 with lastpipe enabled
    # In other shells, this will print 0
    linecnt=0
    printf '%s\n' foo bar | while read -r line
    do
        linecnt=$((linecnt+1))
    done
    echo "total number of lines: $linecnt"

The reason for this potentially surprising behaviour, as described above, is that each SubShell introduces a new variable context and environment. The while loop above is executed in a new subshell with its own copy of the variable linecnt created with the initial value of '0' taken from the parent shell. This copy then is used for counting. When the while loop is finished, the subshell copy is discarded, and the original variable linecnt of the parent (whose value hasn't changed) is used in the echo command.

Different shells exhibit different behaviors in this situation:

• BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').

• BASH creates a new process only if the loop is part of a pipeline.

• KornShell creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. The read example above actually works in ksh88 and ksh93! (but not mksh)

• POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).

More broken stuff:

    # Bash 4
    # The problem also occurs without a loop
    printf '%s\n' foo bar | mapfile -t line  
    printf 'total number of lines: %s\n' "${#line[@]}" # prints 0

    f() {
        if [[ -t 0 ]]; then
            echo "$1"
        else
            read -r var
        fi
    };

    f 'hello' | f
    echo "$var" # prints nothing

Again, in both cases the pipeline causes read or some containing command to run in a subshell, so its effect is never witnessed in the parent process.

• It should be stressed that this issue isn't specific to loops. It's a general property of all pipes, though the "while/read" loop might be considered the canonical example that crops up over and over when people read the help or manpage description of the read builtin and notice that it accepts data on stdin. They might recall that data redirected into a compound command is available throughout that command, but not understand why all the fancy process substitutions and redirects they run across in places like FAQ #1 are necessary. Naturally they proceed to put their funstuff directly into a pipeline, and confusion ensues.

1. Workarounds

• If the input is a file, a simple redirect will suffice:

      # POSIX
      while read -r line; do linecnt=$(($linecnt+1)); done < file
      echo $linecnt

  Unfortunately, this doesn't work with a Bourne shell; see sh(1) from the Heirloom Bourne Shell for a workaround.

• Use command grouping and do everything in the subshell:

      # POSIX
      linecnt=0
      cat /etc/passwd | {
      while read -r line ; do
          linecnt=$((linecnt+1))
      done
      echo "total number of lines: $linecnt"
      }

  This doesn't really change the subshell situation, but if nothing from the subshell is needed in the rest of your code then destroying the local environment after you're through with it could be just what you want anyway.

• Use ProcessSubstitution (Bash only):

      # Bash
      while read -r line; do
          ((linecnt++))
      done < <(grep PATH /etc/profile)
      echo "total number of lines: $linecnt"

  This is essentially identical to the first workaround above. We still redirect a file, only this time the file happens to be a named pipe temporarily created by our process substitution to transport the output of grep.

• Use a named pipe:

      # POSIX
      mkfifo mypipe
      grep PATH /etc/profile > mypipe &
      while read -r line;do
          linecnt=$(($linecnt+1))
      done < mypipe
      echo "total number of lines: $linecnt"

• Use a coprocess (ksh, even pdksh, bash 4, oksh, mksh..):

      # ksh
      grep PATH /etc/profile |&
      while read -r -p line; do
          linecnt=$((linecnt+1))
      done
      echo "total number of lines: $linecnt"

• Use a HereString (Bash only):

       read -ra words <<< 'hi ho hum'
       printf 'total number of words: %d' "${#words[@]}"

  The <<< operator is specific to bash (2.05b and later), however it is a very clean and handy way to specify a small string of literal input to a command.

• With a POSIX shell, or for longer multi-line data, you can use a here document instead:

      # Bash
      declare -i linecnt
      while read -r; do
          ((linecnt++))
      done <<EOF
      hi
      ho
      hum
      EOF
      printf 'total number of lines: %d' "$linecnt"

• Use lastpipe (Bash 4.2)

       # Bash 4.2
       set +m
       shopt -s lastpipe
  
       printf '%s\n' hi{,,,,,} | while read -r "lines[x++]"; do :; done
       printf 'total number of lines: %d' "${#lines[@]}"

  Bash 4.2 introduces the aforementioned ksh-like behavior to Bash. The one caveat is that job control must not be enabled, thereby limiting its usefulness in an interactive shell.

For more related examples of how to read input and break it into words, see FAQ #1.

CategoryShell

How can I access positional parameters after $9?

Use ${10} instead of $10. This works for BASH and KornShell, but not for older BourneShell implementations. Another way to access arbitrary positional parameters after $9 is to use for, e.g. to get the last parameter:

    # Bourne
    for last
    do
        : # nothing
    done

    echo "last argument is: $last"

To get an argument by number, we can use a counter:

    # Bourne
    n=12        # This is the number of the argument we are interested in
    i=1
    for arg
    do
        if test $i -eq $n
        then
            argn=$arg
            break
        fi
        i=`expr $i + 1`
    done
    echo "argument number $n is: $argn"

This has the advantage of not "consuming" the arguments. If this is no problem, the shift command discards the first positional arguments:

    shift 11
    echo "the 12th argument is: $1"

In addition, bash and ksh93 treat the set of positional parameters as an array, and you may use parameter expansion syntax to address those elements in a variety of ways:

    # Bash, ksh93
    for x in "${@:(-2)}"    # iterate over the last 2 parameters
    for y in "${@:2}"       # iterate over all parameters starting at $2
                            # which may be useful if we don't want to shift

Although direct access to any positional argument is possible this way, it's seldom needed. The common alternative is to use getopts to process options (e.g. "-l", or "-o filename"), and then use either for or while to process all the remaining arguments in turn. An explanation of how to process command line arguments is available in FAQ #35, and another is found at http://www.shelldorado.com/goodcoding/cmdargs.html

CategoryShell

How can I randomize (shuffle) the order of lines in a file? (Or select a random line from a file, or select a random file from a directory.)

To randomize the lines of a file, here is one approach. This one involves generating a random number, which is prefixed to each line; then sorting the resulting lines, and removing the numbers.

    #bash
    randomize() {
        while IFS='' read -r l ; do printf "$RANDOM\t%s\n" "$l"; done |
        sort -n |
        cut -f2-
    }

RANDOM is supported by BASH, KornShell but is not defined by posix.

Here's the same idea (printing random numbers in front of a line, and sorting the lines on that column) using other programs:

    # Bourne
    awk '
        BEGIN { srand() }
        { print rand() "\t" $0 }
    ' |
    sort -n |    # Sort numerically on first (random number) column
    cut -f2-     # Remove sorting column

This is (possibly) faster than the previous solution, but will not work for very old AWK implementations (try "nawk", or "gawk", or /usr/xpg4/bin/awk if available). (Note that awk use the epoch time as a seed for srand(), which might not be random enough for you)

A generalized version of this question might be, How can I shuffle the elements of an array? If we don't want to use the rather clumsy approach of sorting lines, this is actually more complex than it appears. A naive approach would give us badly biased results. A more complex (and correct) algorithm looks like this:

    # Uses a global array variable.  Must be compact (not a sparse array).
    # Bash syntax.
    shuffle() {
       local i tmp size max rand

       # $RANDOM % (i+1) is biased because of the limited range of $RANDOM
       # Compensate by using a range which is a multiple of the array size.
       size=${#array[*]}
       max=$(( 32768 / size * size ))

       for ((i=size-1; i>0; i--)); do
          while (( (rand=$RANDOM) >= max )); do :; done
          rand=$(( rand % (i+1) ))
          tmp=${array[i]} array[i]=${array[rand]} array[rand]=$tmp
       done
    }

This function shuffles the elements of an array in-place using the Knuth-Fisher-Yates shuffle algorithm.

Another question we frequently see is, How can I print a random line from a file? The problem here is that you need to know in advance how many lines the file contains. Lacking that knowledge, you have to read the entire file through once just to count them -- or, you have to suck the entire file into memory. Let's explore both of these approaches.

   # Bash
   n=$(wc -l < "$file")        # Count number of lines.
   r=$((RANDOM % n + 1))       # Random number from 1..n. (See below)
   sed -n "$r{p;q;}" "$file"   # Print the r'th line.

   #posix with awk
   awk -v n="$(wc -l<"$file")" 'BEGIN{srand();l=int((rand()*n)+1)} NR==l{print;exit}' "$file"

(see this faq for more info about printing the n'th line.)

The next example sucks the entire file into memory. This approach saves time reopening the file, but obviously uses more memory. (Arguably: on systems with sufficient memory and an effective disk cache, you've read the file into memory by the earlier methods, unless there's insufficient memory to do so, in which case you shouldn't, QED.)

   # Bash
   unset lines i
   while IFS= read -r 'lines[i++]'; do :; done < "$file"   # See FAQ 5
   n=${#lines[@]}
   r=$((RANDOM % n))   # see below
   echo "${lines[r]}"

Note that we don't add 1 to the random number in this example, because the array of lines is indexed counting from 0.

Also, some people want to choose a random file from a directory (for a signature on an e-mail, or to choose a random song to play, or a random image to display, etc.). A similar technique can be used:

    # Bash
    files=(*.ogg)                  # Or *.gif, or *
    n=${#files[@]}                 # For aesthetics
    xmms -- "${files[RANDOM % n]}" # Choose a random element

Note that these last few examples use a simple modulus of the RANDOM variable, so the results are biased. If this is a problem for your application, then use the anti-biasing technique from the Knuth-Fisher-Yates example, above.

Other non portable utilities:

• GNU Coreutils shuf (in recent enough coreutils)

• GNU sort -R

Speaking of GNU coreutils, as of version 6.9 GNU sort has the -R (aka --random-sort) flag. Oddly enough, it only works for the generic locale:

     LC_ALL=C sort -R file     # output the lines in file in random order
     LC_ALL=POSIX sort -R file # output the lines in file in random order
     LC_ALL=en_US sort -R file # effectively ignores the -R option

For more details, see info coreutils sort or an equivalent manual.

• > http://lists.gnu.org/archive/html/bug-bash/2010-01/msg00042.html points out a surprising pitfall concerning the use of RANDOM without a leading $ in certain mathematical contexts. (Upshot: you should prefer n=$((...math...)); ((array[n]++)) over ((array[...math...]++)) in almost every case.)

Behavior described appears reversed in current versions of mksh, ksh93, Bash, and Zsh. Still something to keep in mind for legacy. -ormaaj

CategoryShell

How can two unrelated processes communicate?

Two unrelated processes cannot use the arguments, the environment or stdin/stdout to communicate; some form of inter-process communication (IPC) is required.

1. A file

Process A writes in a file, and Process B reads the file. This method is not synchronized and therefore is not safe if B can read the file while A writes in it. A lockdir or a signal can probably help.

2. A directory as a lock

mkdir can be used to test for the existence of a dir and create it in one atomic operation; it thus can be used as a lock, although not a very efficient one.

Script A:

    until mkdir /tmp/dir;do # wait until we can create the dir
      sleep 1
    done
    echo foo > file         # write in the file this section is critical
    rmdir /tmp/dir          # remove the lock

Script B:

    until mkdir /tmp/dir;do #wait until we can create the dir
      sleep 1
    done
    read var < file         # read in the file this section is, critical
    echo "$var"             # Script A cannot write in the file
    rmdir /tmp/dir          # remove the lock

See Faq #45 and mutex for more examples with a lock directory.

3. Signals

Signals are probably the simplest form of IPC:

ScriptA:

    trap 'flag=go' USR1 #set up the signal handler for the USR1 signal

    # echo $$ > /tmp/ScriptA.pid #if we want to save the pid in a file

    flag=""
    while [[ $flag != go ]]; do # wait for the green light from Script B
      sleep 1;
    done
    echo we received the signal

You must find or know the pid of the other script to send it a signal using kill:

     #kill all the
     pkill -USR1 -f ScriptA 
     
     #if ScriptA saved its pid in a file
     kill -USR1 $(</var/run/ScriptA.pid)

     #if ScriptA is a child:
     ScriptA & pid=$!
     kill -USR1 $pid

The first 2 methods are not bullet proof and will cause trouble if you run more than one instance of scriptA.

4. Named Pipes

Named pipes are a much richer form of IPC. They are described on their own page: NamedPipes.

CategoryShell

How do I determine the location of my script? I want to read some config files from the same place.

This topic comes up frequently. This answer covers not only the expression used above ("configuration files"), but also several variant situations. If you've been directed here, please read this entire answer before dismissing it.

This is a complex question because there's no single right answer to it. Even worse: it's not possible to find the location reliably in 100% of all cases. All ways of finding a script's location depend on the name of the script, as seen in the predefined variable $0. But providing the script name in $0 is only a (very common) convention, not a requirement.

The suspect answer is "in some shells, $0 is always an absolute path, even if you invoke the script using a relative path, or no path at all". But this isn't reliable across shells; some of them (including BASH) return the actual command typed in by the user instead of the fully qualified path. And this is just the tip of the iceberg!

Your script may not actually be on a locally accessible disk at all. Consider this:

  ssh remotehost bash < ./myscript

The shell running on remotehost is getting its commands from a pipe. There's no script anywhere on any disk that bash can see.

Moreover, even if your script is stored on a local disk and executed, it could move. Someone could mv the script to another location in between the time you type the command and the time your script checks $0. Or someone could have unlinked the script during that same time window, so that it doesn't actually have a link within a file system any more.

Even in the cases where the script is in a fixed location on a local disk, the $0 approach still has some major drawbacks. The most important is that the script name (as seen in $0) may not be relative to the current working directory, but relative to a directory from the program search path $PATH (this is often seen with KornShell). Or (and this is most likely problem by far...) there might be multiple links to the script from multiple locations, one of them being a simple symlink from a common PATH directory like /usr/local/bin, which is how it's being invoked. Your script might be in /opt/foobar/bin/script but the naive approach of reading $0 won't tell you that -- it may say /usr/local/bin/script instead.

(For a more general discussion of the Unix file system and how symbolic links affect your ability to know where you are at any given moment, see this Plan 9 paper.)

Having said all that, if you still want to make a whole slew of naive assumptions, and all you want is the fully qualified version of $0, you can use something like this (BASH syntax):

  [[ $0 == /* ]] && echo "$0" || echo "${PWD}/${0#./}"

Or the BourneShell version:

  case $0 in /*) echo "$0";; *) echo "`pwd`/$0";; esac

Or a shell-independent variant (needs a readlink(1) supporting -f, though, so it's OS-dependent):

  readlink -f "$0"

In Bash, version 4.1.7(1)-release, on Linux, it seems bash always opens the script with fd 255 so you can just do:

  HOME="$(dirname "$(readlink /proc/$$/fd/255)")"

If we want to account for the cases where the script's relative pathname (in $0) may be relative to a $PATH component instead of the current working directory (as mentioned above), we can try to search for the script in all the directories of $PATH.

The following script shows how this could be done:

   1 #!/bin/bash
   2 
   3 myname=$0
   4 if [[ -s "$myname" ]] && [[ -x "$myname" ]]; then
   5     # $myname is already a valid file name
   6 
   7     mypath=$myname
   8 else
   9     case "$myname" in
  10     /*) exit 1;;             # absolute path - do not search PATH
  11     *)
  12         # Search all directories from the PATH variable. Take
  13         # care to interpret leading and trailing ":" as meaning
  14         # the current directory; the same is true for "::" within
  15         # the PATH.
  16 
  17         # Replace leading : with . in PATH, store in p
  18         p=${PATH/#:/.:}
  19         # Replace trailing : with .
  20         p=${p//%:/:.}
  21         # Replace :: with :.:
  22         p=${p//::/:.:}
  23         # Temporary input field separator, see FAQ #1
  24         OFS=$IFS IFS=:
  25         # Split the path on colons and loop through each of them
  26         for dir in $p; do
  27                 [[ -f "$dir/$myname" ]] || continue # no file
  28                 [[ -x "$dir/$myname" ]] || continue # not executable
  29                 mypath=$dir/$myname
  30                 break           # only return first matching file
  31         done
  32         # Restore old input field separator
  33         IFS=$OFS
  34         ;;
  35     esac
  36 fi
  37 
  38 if [[ ! -f "$mypath" ]]; then
  39     echo >&2 "cannot find full path name: $myname"
  40     exit 1
  41 fi
  42 
  43 echo >&2 "path of this script: $mypath"

Note that $mypath is not necessarily an absolute path name. It still can contain relative paths like ../bin/myscript, because $PATH could contain those. If you want to get the directory only from that string, check FAQ 73.

Are you starting to see how ridiculously complex this problem is becoming? And this is still just the simplistic case where we've made a lot of assumptions about the script not moving and not being piped in!

Generally, storing data files in the same directory as their programs is a bad practise. The Unix file system layout assumes that files in one place (e.g. /bin) are executable programs, while files in another place (e.g. /etc) are data files. (Let's ignore legacy Unix systems with programs in /etc for the moment, shall we....)

Here are some common sense alternatives you should consider, instead of attempting to perform the impossible:

• It really makes the most sense to keep your script's configuration in a single, static location such as /etc/foobar.conf.

• If you need to define multiple configuration files, then you can have a directory (say, /var/lib/foobar/ or /usr/local/lib/foobar/), and read that directory's location from a fixed place such as /etc/foobar.conf.

• If you don't even want that much to be hard-coded, you could pass the location of foobar.conf (or of your configuration directory itself) as a parameter to the script.

• If you need the script to assume certain default in the absence of /etc/foobar.conf, you can put defaults in the script itself, or fall back to something like $HOME/.foobar.conf if /etc/foobar.conf is missing.

• When you install the script on a target system, you could put the script's location into a variable in the script itself. The information is available at that point, and as long as the script doesn't move, it will always remain correct for each installed system.
• In most cases, it makes more sense to abort gracefully if your configuration data can't be found by obvious means, rather than going through arcane processes and possibly coming up with wrong answers.

  • BASH_SOURCE is probably a much better idea than $0, gives better results and is better defined. This article should probably be rewritten with BASH_SOURCE in mind. --Lhunath

CategoryShell

How can I display the target of a symbolic link?

The nonstandard external command readlink(1) can be used to display the target of a symbolic link:

$ readlink /bin/sh
bash

If you don't have readlink, you can use Perl:

perl -e 'print readlink "/bin/sh", "\n"'

You can also use GNU find's -printf %l directive, which is especially useful if you need to resolve links in batches:

$ find /bin/ -type l -printf '%p points to %l\n'
/bin/sh points to bash
/bin/bunzip2 points to bzip2
...

If your system lacks both readlink and Perl, you can use a function like this one:

# Bash
readlink() {
    local path=$1 ll

    if [ -L "$path" ]; then
        ll=$(LC_ALL=C ls -l "$path" 2>/dev/null) &&
        printf '%s\n' "${ll#* -> }"
    else
        return 1
    fi
}

However, this can fail if a symbolic link contains " -> " in its name.

CategoryShell

How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?

Some GNU/Linux distributions have a rename(1) command, which you can use for the former; however, the syntax differs from one distribution to the next, so it's not a portable answer....

Consult your system's man pages if you want to learn how to use your rename command, if you have one at all. It's often perfectly good for one-shot interactive renames, just not in portable scripts. We don't include any rename examples here because it's too confusing -- there are two common versions of it and they're totally incompatible with each other.

You can do non-recursive mass renames portably with a loop and some Parameter Expansions, like this:

# POSIX
# Rename all *.foo to *.bar
for f in *.foo; do mv -- "$f" "${f%.foo}.bar"; done

# POSIX
# This removes the extension .zip from all the files.
for file in ./*.zip; do mv "$file" "${file%.zip}"; done

The "--" and "./*" are to protect from problematic filenames that begin with "-". You only need one or the other, not both, so pick your favorite.

Here are some similar examples, using Bash-specific parameter expansions:

# Bash
# Replace all spaces with underscores
for f in *\ *; do mv -- "$f" "${f// /_}"; done

# Bash
# Rename all "foo" to "bar"
for file in ./*foo*; do mv "$file" "${file//foo/bar}"; done

All the above examples invoke the external command mv(1) once for each file, so they may not be as efficient as some of the rename implementations.

If you want to rename files recursively, then it becomes much more challenging. This example renames *.foo to *.bar:

# Bash
# Also requires GNU or BSD find(1)
# Recursively change all *.foo files to *.bar

find . -type f -name '*.foo' -print0 | while IFS= read -r -d '' f; do
  mv -- "$f" "${f%.foo}.bar"
done

This example uses Bash 4's globstar instead of GNU find:

# Bash 4 which requires globstar to be enabled. NOT portable!
# Rename all "foo" files to "bar" recursively.
# "foo" must NOT appear in a directory name.

shopt -s globstar; for file in /path/to/**/*foo*; do mv -- "$file" "${file//foo/bar}"; done

To check what will be the output of the above command, you can add an echo before the mv so you will get an idea.

For more techniques on dealing with files with inconvenient characters in their names, see FAQ #20.

The trickiest part of recursive renames is ensuring that you do not change the directory component of a pathname, because something like this is doomed to failure:

mv "./FOO/BAR/FILE.TXT" "./foo/bar/file.txt"

Therefore, any recursive renaming command should only change the filename component of each pathname. If you need to rename the directories as well, those should be done separately. Furthermore, recursive directory renaming should either be done depth-first (changing only the last component of the directory name in each instance), or in several passes. Depth-first works better in the general case.

Here's an example script that uses depth-first recursion (changes spaces in names to underscores, but you just need to change the ren() function to do anything you want) to rename both files and directories (again, it's easy to modify to make it act only on files or only on directories, or to act only on files with a certain extension, to avoid or force overwriting files, etc.):

# Bash
ren() {
  local newname
  newname=${1// /_}
  [ "$1" != "$newname" ] && mv -- "$1" "$newname"
}

traverse() {
  local i
  cd -- "$1" || exit 1
  for i in *; do
    [ -d "$i" ] && traverse "$i"
    ren "$i"
  done
  cd .. || exit 1
}

# main program
shopt -s nullglob
traverse /path/to/startdir

Here is another way to recursively rename all directories and files with spaces in their names:

find . -depth -name "* *" -exec bash -c 'dir=${1%/*} base=${1##*/}; mv "$1" "$dir/${base// /_}"' _ {} \;

or, if your version of find accepts it, this is more efficient as it runs one bash for many files instead of one bash per file:

find . -depth -name "* *" -exec bash -c 'for f; do dir=${f%/*} base=${f##*/}; mv "$f" "$dir/${base// /_}"; done' _ {} +

To convert filenames to lower case, if you have the utility mmv(1) on your machine, you could simply do:

# convert all filenames to lowercase
mmv "*" "#l1"

Otherwise, you need something that can take a mixed-case filename as input and give back the lowercase version as output. In Bash 4 and higher, there is a parameter expansion that can do it:

# Bash 4
for f in *[[:upper:]]*; do mv -- "$f" "${f,,*}"; done

Otherwise, tr(1) may be helpful:

# tolower - convert file names to lower case
# POSIX
for file in "$@"
do
    [ -f "$file" ] || continue                # ignore non-existing names
    newname=$(echo "$file" | tr '[:upper:]' '[:lower:]')     # lower case
    [ "$file" = "$newname" ] && continue      # nothing to do
    [ -f "$newname" ] && continue             # don't overwrite existing files
    mv -- "$file" "$newname"
done

We use the fancy range notation, because tr can behave very strangely when using the A-Z range on some locales:

imadev:~$ echo Hello | tr A-Z a-z
hÉMMÓ

To make sure you aren't caught by surprise when using tr with ranges, either use the fancy range notations, or set your locale to C.

imadev:~$ echo Hello | LC_ALL=C tr A-Z a-z
hello
imadev:~$ echo Hello | tr '[:upper:]' '[:lower:]'
hello
# Either way is fine here.

This technique can also be used to replace all unwanted characters in a file name, e.g. with '_' (underscore). The script is the same as above, with only the "newname=..." line changed.

# renamefiles - rename files whose name contain unusual characters
# POSIX
for file in "$@"
do
    [ -f "$file" ] || continue            # ignore non-regular files, etc.
    newname=$(echo "$file" | sed 's/[^[:alnum:]_.]/_/g')
    [ "$file" = "$newname" ] && continue  # nothing to do
    [ -f "$newname" ] && continue         # do not overwrite existing files
    mv -- "$file" "$newname"
done

The character class in [] contains all the characters we want to keep (after the ^); modify it as needed. The [:alnum:] range stands for all the letters and digits of the current locale.

Here's an example that does the same thing, but this time using Parameter Expansion instead of sed:

# renamefiles (more efficient, less portable version)
# Bash
for file in "$@"; do
   [ -f "$file" ] || continue
   newname=${f//[^[:alnum:]_.]/_}
   [ "$file" = "$newname" ] && continue
   [ -f "$newname" ] && continue
   mv -- "$file" "$newname"
done

It should be noted that all these examples contain a race condition -- an existing file could be overwritten if it is created in between the [ -f "$newname" ... and mv "$file" ... commands. Solving this issue is beyond the scope of this page, however.

One final note about changing the case of filenames: when using GNU mv, on many file systems, attempting to rename a file to its lowercase or uppercase equivalent will fail. (This applies to Cygwin on DOS/Windows systems using FAT or NTFS file systems; to GNU mv on Mac OS X systems using HFS+ in case-insensitive mode; as well as to Linux systems which have mounted Windows/Mac file systems, and possibly many other setups.) GNU mv checks both the target names before attempting a rename, and due to the file system's mapping, it thinks that the destination "already exists":

mv README Readme    # fails with GNU mv on FAT file systems, etc.

The workaround for this is to rename the file twice: first to a temporary name which is completely different from the original name, then to the desired name.

mv README tempfilename &&
mv tempfilename Readme

CategoryShell

What is the difference between test, [ and [[ ?

[ ("test" command) and [[ ("new test" command) are used to evaluate expressions. [[ works only in Bash and Korn shell, and is more powerful; [ and test are available in POSIX shells. Here are some examples:

    if [ -z "$variable" ]
    then
        echo "variable is empty!"
    fi

    if [ ! -f "$filename" ]
    then
        echo "not a valid, existing file name: $filename"
    fi

and

    if [[ ! -e $file ]]
    then
        echo "directory entry does not exist: $file"
    fi

    if [[ $file0 -nt $file1 ]]
    then
        echo "file $file0 is newer than $file1"
    fi

To cut a long story short: test implements the old, portable syntax of the command. In almost all shells (the oldest Bourne shells are the exception), [ is a synonym for test (but requires a final argument of ]). Although all modern shells have built-in implementations of [, there usually still is an external executable of that name, e.g. /bin/[. POSIX defines a mandatory feature set for [, but almost every shell offers extensions to it. So, if you want portable code, you should be careful not to use any of those extensions.

[[ is a new improved version of it, and is a keyword, not a program. This makes it easier to use, as shown below. [[ is understood by KornShell and BASH (e.g. 2.03), but not by the older POSIX or BourneShell.

Although [ and [[ have much in common, and share many expression operators like "-f", "-s", "-n", "-z", there are some notable differences. Here is a comparison list: