Wednesday, January 27, 2010

Sed - save changes to same file


Sed receives text input, either from stdin or from a file, performs certain operations on specified lines(or all lines) of the input, one line at a time, then outputs the result to stdout or to a file. Today I am going to show how we can use sed to do some operation on a file (mainly substitute, which is the most popular with sed) and write back the results to the same file.

Input file:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

Lets try to replace the word 'VERSION' in the above file with '8.04'

$ sed 's/VERSION/8.04/' file.txt
port:9903
os-version:8.04
codename:hardy
status:active

So, be default sed outputs the result to 'stdout'.

Append or redirection to the same filename will be wrong !!

$ sed 's/VERSION/8.04/' file.txt > file.txt


Newer sed versions (e.g sed version 4.1.4), there is a useful command line option:

-i[SUFFIX], --in-place[=SUFFIX]

Description: edit files in place (makes backup if extension supplied)

Lets try this option:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed -i 's/VERSION/8.04/' file.txt

$ cat file.txt
port:9903
os-version:8.04
codename:hardy
status:active

It worked; the result is printed to the same filename.

We can also mention the backup extension like this:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed -i.bak 's/VERSION/8.04/' file.txt

$ cat file.txt
port:9903
os-version:8.04
codename:hardy
status:active

The original content of the input file is backed up here:

$ cat file.txt.bak
port:9903
os-version:VERSION
codename:hardy
status:active

With older version of 'sed' editor (where this -i option is absent), we can write the result to a temporary file and then in the next step we can move the temporary file back to the original file like this:

$ cat file.txt
port:9903
os-version:VERSION
codename:hardy
status:active

$ sed 's/VERSION/8.04/' file.txt > file.txt.tmp
$ mv file.txt.tmp file.txt

And to work with more number of files (say perform the same replacement as above in all the .cfg files in current directory, including sub-directory)

for file in $(find . -name "*.cfg")
do
echo "Replacing on : $file"
sed 's/VERSION/8.04/' $file > $file.tmp
mv $file.tmp $file
echo "Replacement done on : $file"
done

Related posts:

- Add Change Insert lines to file using sed
- Substitute character by position using sed
- Case insensitive search and replace using sed
- Accessing external variables in sed and awk
- Delete next few lines using sed

5 comments:

Stu said...

Hey, nice blog, I get a lot of enjoyment out of reading random snippets like these.

I noticed you didn't quote $file though:

for file in $(find . -name "*.cfg")
do
echo "Replacing on : $file"
sed 's/VERSION/8.04/' $file > $file.tmp
mv $file.tmp $file
echo "Replacement done on : $file"
done


This will break if $file ever has spaces or other abnormal characters.

probably just an oversight, but hey I'm in the mood to rant...

http://bash-hackers.org/wiki/doku.php/syntax/words
http://www.grymoire.com/Unix/Quote.html

Jadu Saikia said...

@Stu, thanks for the same.

obakesan said...

agreed ... nice to see there are still other command line dweebs out there (aside from me)

Karan Bohra said...

This method is immune from filenames with spaces, newlines, etc.
We use "find" to select the relevant files that "sed" can operate
upon.

find . -name '*.cfg' -o -name '.*.cfg' -type f ! -size 0 -perm -u=rw -exec sh -c '
shift "$1"; _d=$(mktemp -d);
while case $# in 0) break;; esac; do;
file -b "$1" | egrep -qw -e "ASCII" -e "text" || {
echo >&2 "Warning: \"$1\" is not a text file... Skipping.";
shift; continue;
}
_f=$1; shift; status=0
echo "Replacing on: $_f";
sed -e "s/VERSION/8.04/" < $_f > $_d/_f && \
mv -f "$_d/_f" "$_f" || status=$?
case $status in
0) echo "Replacement on: \"$_f\" ....OK.";;
*) echo "Replacement on: \"$_f\" ...NOK.";;
esac
done;
rm -rf "$_d";
' 2 1 {} +

Karan Bohra said...

The "find" command is missing parentheses due to which it will give incorrect results.

> find . -name '*.cfg' -o -name '.*.cfg' -type f ! -size 0 -perm -u=rw -exec sh -c

We need to enclose the -name options within parens and also escape the parens so
that they are not intercepted by the shell before "find" gets a chance to look at them.

find . \( -name '*.cfg' -o -name '.*.cfg' \) -type f ! -size 0 -perm -u=rw -exec sh -c

It's better recast as :

find . \
\( -name '*.cfg' -o -name '.*.cfg' \) \
-type f \
! -size 0 \
-perm -u=rw \
-exec sh -c '
# your bash code here...
' 2 1 {} +

and now we can easily read the intent of "find", viz.,

i) grab any entry in the current dir & below which has a name ending
in .cfg including a hidden file as well.

ii) entry just selected ought to be a plain file.

iii) the plain file selected should not be size zero.

iv) nozero sized file should be readable & writable by you the user.

v) finally, collect a bunch of the files selected above and feed them to bash.
this helps generate fewer calls to bash.

© Jadu Saikia www.UNIXCL.com