UNIX Command Line

$PATH on my ubuntu:

$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/org/bin:/opt/appls


Now to remove any path which contain "/org/" from PATH variable:

$ echo $PATH | tr ':' '\n' | awk '$0 !~ "/org/"' | paste -sd:
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appls

Export the above to PATH

$ export PATH=$(echo $PATH | tr ':' '\n' | awk '$0 !~ "/org/"' | paste -sd:)
$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appls

Use
awk '$0 != "/usr/local/sbin"'
in the above export awk part if you exactly want to remove the path "/usr/local/sbin" from PATH.

And using bash array:

$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/org/bin:/opt/appl

Remove any path which contain "/org/" from PATH

The one liner:
$ PATH=$(IFS=':';p=($PATH);unset IFS;p=(${p[@]%%*/org/*});IFS=':';echo "${p[*]}";unset IFS)

$ echo $PATH
o/p:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/appl

Input file:

$ cat details.txt
May 27 17:01:07 IST 2009
A,5050,3434
B,42122,12121
C,26232,3424
May 27 18:01:47 IST 2009
A,1212,3434
B,12122,12121
C,23232,24324
May 27 19:01:27 IST 2009
A,8212,6434
B,2122,2121
C,3232,4324

Required output: Divide the above file into sub-files ( based on the codes i.e. A,B,C etc) and each sub-file should contain the information related to that code (Basically the lines which starts with this code) and the sub-file should look like:

$ cat A.out
May 27 17:01:07 IST 2009,5050,3434
May 27 18:01:47 IST 2009,1212,3434
May 27 19:01:27 IST 2009,8212,6434

$ cat B.out
May 27 17:01:07 IST 2009,42122,12121
May 27 18:01:47 IST 2009,12122,12121
May 27 19:01:27 IST 2009,2122,2121

$ cat C.out
May 27 17:01:07 IST 2009,26232,3424
May 27 18:01:47 IST 2009,23232,24324
May 27 19:01:27 IST 2009,3232,4324

Awk script for the same:

$ awk '
    BEGIN {FS=OFS=","}
    $1 ~ /IST/ {dt=$0; next}
    {print dt,$2,$3 > $1".out"}
' details.txt

Related posts:
- Divide a file using awk
- Split file into subfiles based on pattern using awk
- Append line identifier to each line using awk

$ line="abcdefghij"

$ echo $line
abcdefghij

#Number of characters in the line
$ echo ${#line}
10

#No of characters - 4
$ echo $((${#line}-4))
6

#Removing last 4 characters (The value 6 below is obtained from above expression)
$ echo ${line::6}
abcdef

#Printing last 4 characters
$ echo ${line: -4}
ghij

With this reference lets try to remove last 4 charactes from all lines of the following input file:

$ cat details.txt
some junk chrs
some more junks
pull some more chrs
this is just an example

The bash script(On prompt):

$ while read line
    do
         echo ${line::$((${#line}-4))}
done < details.txt

Output:
some junk
some more j
pull some more
this is just an exa

Related post:
- Bash script to find its own location
- Remove last two characters using awk
- Print last two characters using awk and sed
- Bash parameter substitution is well explained here and here

Question: How can a bash script determine it's own location ?

Lets explore this using the following example script.

The script "mypath.sh" resides under my "~/work/engine/"(i.e. /home/user1/work/engine/) directory.

$ ls -l ~/work/engine/
total 4
-rwxr-xr-x 1 user1 users 239 2009-05-25 22:15 mypath.sh

Have a look on the script: (The main intention of the script to print the location of the script mypath.sh and create some three directories named log,tmp,conf under the directory where this script resides)

$ cat ~/work/engine/mypath.sh
#!/bin/sh

echo "Value of \$0 is : $0"

echo "path to script : ${0%/*}"
apath=$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")
echo "absolutepath is : $apath"

dname=$(dirname $apath)
echo "directory name : $dname"

mkdir -p $dname/{log,tmp,conf}

We will try to execute this script from "/home/user1/work" directory.

$ pwd
/home/user1

$ cd work/

Execute the script:

$ engine/mypath.sh

output:

Value of $0 is : engine/mypath.sh
path to script : engine
absolutepath is : /home/user1/work/engine/mypath.sh
directory name : /home/user1/work/engine

Lets confirm the creation of the directories.

$ ls -l engine/

total 16
drwxr-xr-x 2 user1 users 4096 2009-05-25 22:15 conf
drwxr-xr-x 2 user1 users 4096 2009-05-25 22:15 log
-rwxr-xr-x 1 user1 users  239 2009-05-25 22:15 mypath.sh
drwxr-xr-x 2 user1 users 4096 2009-05-25 22:15 tmp

Requirement:

To move first 3 .log files to the "bak" directory.

The ways:

$ ls *.log | sed '1,3 !d' | xargs -i mv {} bak/

Same as

$ ls *.log | sed -n '1,3 p' | xargs -i mv {} bak/

Same as

$ ls *.log | head -3 | xargs -i mv {} bak/

Same as

$ ls *.log | head -3 | while read file
   do
       mv $file bak/
done

And using bash array:

$ FILES=(*.log)
$ mv "${FILES[@]:0:3}" bak/

Well it sounds very simple, but thought would be useful for all vi newbies :-)

This is how we can remove all leading white space(s) from all lines in vi editor:

In ex mode:

:1,$ s/^\s*//g

Related post:

- Delete blank lines in vi editor
- More vi editor tips post from unstableme here

Input file:

$ cat details.txt
1|Manager1|sw1
2|Manager3|sw5
3|Manager1|sw4
4|Manager2|sw9
5|Manager2|sw12
6|Manager1|sw2
7|Manager1|sw0

Required: Remove the duplicate lines from the above file for which the occurrence of 2nd field is consecutive.

The following awk one liner going to remove the duplicate lines based on consecutive 2nd field (is going to keep first occurrence)

$ awk 'x !~ $2; {x=$2}' FS=\| details.txt

o/p:

1|Manager1|sw1
2|Manager3|sw5
3|Manager1|sw4
4|Manager2|sw9
6|Manager1|sw2

And to remove duplicate, nonconsecutive fields

$ awk '!arr[$2]++' FS=\| details.txt

o/p:

1|Manager1|sw1
2|Manager3|sw5
4|Manager2|sw9

Related posts:
- Remove duplicate based on fields in awk
- Remove duplicate without sorting file - awk
- Remove duplicate blank lines using awk in bash

Input file:

$ cat file.txt
ID502,
Kia,
class v,
pass

ID506,
Nill,
class v,
fail
ID902,
Herry,
class vi,
pass

Output required:
Combine the consecutive lines related to a particular student ID (^ID field) in single line. i.e. required output:

ID502,Kia,class v,pass
ID506,Nill,class v,fail
ID902,Herry,class vi,pass

Awk solution:

$ awk 'NR > 1 && /^ID/{print ""}
 {printf $0}
 END {print ""}
' file.txt

Related post:

- Combine every 3 lines as one line using awk
- Merging lines using awk in bash
- Split a file into sub-files based on start pattern - awk

Thought of continuing a similar post (w.r.t. few of my recent posts) based on awk array.

Input file:

$ cat details.txt
Manager1|sw1
Manager3|sw5
Manager1|sw4
Manager2|sw9
Manager2|sw12
Manager1|sw2
Manager1|sw0

Output required:
Group the similar (based on $1) fields($2) together, i.e. group the engineers which are under a particular common manager. i.e. required output:

Manager1|sw1,sw4,sw2,sw0
Manager2|sw9,sw12
Manager3|sw5

Awk solution:

$ awk '
   BEGIN {FS=OFS="|"}
   !A[$1] {A[$1] = $0; next}
   {A[$1] = A[$1] "," $2}
   END {for(i in A) {print A[i]}
}' details.txt

Lets add one more field as the "team" field. e.g. "Manager1" manages engineer "sw1" which is from team1.

$ cat details1.txt
Manager1|team1|sw1
Manager3|team4|sw5
Manager1|team2|sw4
Manager2|team5|sw9
Manager2|team5|sw12
Manager1|team3|sw2
Manager1|team2|sw0

Now lets try to group the engineers which are from the same team and are being managed by the same common manager. The awk solution would be:

$ awk '
   BEGIN {FS=OFS="|"}
   !A[$1$2] {A[$1$2] = $0; next}
   {A[$1$2] = A[$1$2] "," $3}
   END {for(i in A) {print A[i]}
}' details1.txt

o/p:

Manager1|team1|sw1
Manager1|team2|sw4,sw0
Manager1|team3|sw2
Manager3|team4|sw5
Manager2|team5|sw9,sw12

Graphically my requirement was like this :


I had to append a field before each line based on the 1st field (ID field), i.e. for the first line for a specific unique ID, the field with text "Agg line" should be appended and for the successive lines for that ID, field with text "sub-line" has to be appended.

So my input file was in a csv format like this:

$ cat details.txt
ID5,17.95,107.0,Y
ID5,6.56,12.3,Y
ID5,7.36,22.5,Y
ID5,4.03,72.2,Y
ID6,282.8,134.1,Y
ID6,111.56,61.7,Y
ID6,171.24,72.4,Y
ID7,125.6,89,Y

Output required:

Agg line,ID5,17.95,107.0,Y
sub-line,ID5,6.56,12.3,Y
sub-line,ID5,7.36,22.5,Y
sub-line,ID5,4.03,72.2,Y
Agg line,ID6,282.8,134.1,Y
sub-line,ID6,111.56,61.7,Y
sub-line,ID6,171.24,72.4,Y
Agg line,ID7,125.6,89,Y

Awk solution:

$ awk '
    BEGIN {FS=OFS=","}
        !_[$1]++{print "Agg line",$0;next}
        {print "sub-line",$0}
' details.txt

Related posts:

- Replace field other than the first occurrence using awk

This post is mainly for awk newbies.

Input file:

$ cat details.txt
area=55
172.22.23.5
172.22.23.6
172.22.23.7
area=54
172.16.1.96
172.16.1.97
172.16.1.98
area=60
172.21.5.12
172.21.5.13

Output required: Append the line identifier i.e. the "area id" to each of the related ips) i.e. required output:

172.22.23.5,55
172.22.23.6,55
172.22.23.7,55
172.16.1.96,54
172.16.1.97,54
172.16.1.98,54
172.21.5.12,60
172.21.5.13,60

Awk solutions:

$ awk -F "=" '
BEGIN { OFS="," }
{
  if ($1 == "area") {
      AID=$2
}   else {
      print $0,AID
  }
}
'  details.txt

or

$ awk -F "=" '
BEGIN { OFS="," }
  /^area/ {AID=$2;next}
{print $0,AID}
' details.txt

or

$ awk -F "=" '
BEGIN { OFS="," }
  $1=="area" {AID=$2;next}
{print $0,AID}
' details.txt

Input file:

$ cat file.txt
"12"|"w2432"|"awk"
"22"|"35435"|"sed"
"13.2"|"adad"|"awk"
"9"|""qqwq"|"perl"

Output required: Print only those lines where first field value is > 12

Way1:
Normal $2 > 12 condition is not going to work as $2 is with "". So we need to remove the double quotes from first field and then we can compare the condition.

$ awk -v c='"' 'BEGIN {
FS=OFS="|"
}
{
f=$1
gsub(c,"",f)
if (f > 12) print
}
' file.txt

o/p:

"22"|"35435"|"sed"
"13.2"|"adad"|"awk"

Way2:
Another way would be to use 2 field separator (FS) with awk:

$ awk  -F "[\",|]" '$2 > 12' file.txt

o/p:

"22"|"35435"|"sed"
"13.2"|"adad"|"awk"

Input file master-classVI.txt contains some details of class VI students.

$ cat master-classVI.txt
sachin:22:M
rupali:12:F
nilutpal:11:M
rohan:01:M
wasim:19:M
priya:08:F
monali:44:F
rashi:11:F

Results of unit test on English language has come in results-fall.txt (it contains the name of all passed students)

$ cat results-fall.txt
priya
rashi
sachin
nilutpal

Required output:
Modify master-classVI.txt with the result. i.e. pass/fail as the last column. i.e. required output

sachin:22:M:Pass
rupali:12:F:Fail
nilutpal:11:M:Pass
rohan:01:M:Fail
wasim:19:M:Fail
priya:08:F:Pass
monali:44:F:Fail
rashi:11:F:Pass

Awk solution using awk NR FNR variables

$ awk 'BEGIN {FS=OFS=":"}
    {$4="Fail"}
    FNR==NR{_[$1]=$1;next}
        $1 in _{$4="Pass"} {print}
' results-fall.txt master-classVI.txt

Some explanation:

a) By default make $4 = "Fail" ; all failed huh !!
b) I have put a lot of posts on awk NR FNR ; just for you understanding print this:

$ awk '
{print FNR,NR,"["$0"]"}' results-fall.txt master-classVI.txt

1 1 [priya]
2 2 [rashi]
3 3 [sachin]
4 4 [nilutpal]
1 5 [sachin:22:M]
2 6 [rupali:12:F]
3 7 [nilutpal:11:M]
4 8 [rohan:01:M]
5 9 [wasim:19:M]
6 10 [priya:08:F]
7 11 [monali:44:F]
8 12 [rashi:11:F]

c) So array _ is going to contain "priya","rashi","sachin","nilutpal"
Records from "results-fall.txt" is avoided from printing using the 'next' above.

d) For all the above entries in the array _ , make $4 "Pass"

Input data.txt is a collection report for XYZ corp group by different collection zones.

$ cat data.txt
Total Collection = $10291 {Fri May  8}
zone7   4500
zone8   3545
zone1   1200
zone0   900
zone3   70
zone5   67
zone11  9
Total Collection = $11847 {Sat May  9}
zone1   2800
zone3   2800
zone6   2567
zone8   2300
zone9   1200
zone12  90
zone11  90

Required: We need to find out the top 4 collection zones for each day from the above file. i.e. to print next 4 lines where the pattern "Total Collection =" is found (as the items are sorted on collection amount).
This is how we can achieve this using awk:

$ awk '/^Total Collection =/{c=4;next}c-->0' data.txt

zone7   4500
zone8   3545
zone1   1200
zone0   900
zone1   2800
zone3   2800
zone6   2567
zone8   2300

Now if we need to print the header line also, something like:

$ awk '/^Total Collection =/{c=4;{print}next}c-->0' data.txt

Total Collection = $10291 {Fri May  8}
zone7   4500
zone8   3545
zone1   1200
zone0   900
Total Collection = $11847 {Sat May  9}
zone1   2800
zone3   2800
zone6   2567
zone8   2300

And if you want to just print the date part as the header with top 4 collection zones.

$ awk -F "[{,}]" '/^Total Collection =/{c=4;{print $2}next}c-->0' data.txt

Fri May  8
zone7   4500
zone8   3545
zone1   1200
zone0   900
Sat May  9
zone1   2800
zone3   2800
zone6   2567
zone8   2300

As you can see we have used multiple field separator(FS) above to extract the date part from the header lines. A post on multiple FS with awk can be found here

Related posts:
- Print current,previous,next line where pattern is matched using awk and sed here
- Delete next few lines after pattern is found using sed here
- Print first line of each paragraph using awk here

Input file:

$ cat file.txt
S Badrinath
c Singh b Pathan
0

ML Hayden
c Jayawardene b Sreesanth
89
6X6

SK Raina
c Goel b Chawla
32
1X6

Required: Print the first line of each paragraph (each paragraph being separated by a single new line)
Awk solution:

$ awk 'NR==1||cnt-->0;/^$/{cnt=1}' file.txt

Output:

S Badrinath
ML Hayden
SK Raina

Related post:
- Make 4 line paragraph using awk here
- Make 4 line paragraph using Linux paste command here

By default, Linux remembers the sudo password for some time. i.e. if you run a command using sudo command, first time it will ask you for sudo password, but if you sudo it again within some seconds, Linux is not going to prompt you for sudo password.

If you are worried about this due to any security concern or so and you want Linux to prompt for sudo password every time you use sudo to run a command, the following can be done.

$ sudo visudo

Enter the following in the default section

Defaults    timestamp_timeout = 0

and save the file and you are done.

** The value 0 in the above line can be replaced with any "minutes" for which sudo password has to be remembered.

Input files:
main.txt is the master file with some details in the order
country-code|epoch|name|value

And id.txt is a look-up table which contains the country-code to country-name mapping.

$ cat main.txt
512|1241503759|ax|90
234|1241503760|ay|10
122|1241503823|az|90
123|1241503947|at|80

$ cat id.txt
122|US
123|IN
125|NZ
234|HK
512|ZM
600|KR

Required:
Country codes(field 1) in main.txt has to be replaced with the corresponding country-name from look-up table id.txt

Solutions: Normal bash scripting would be:

$ cat main.txt | while read line
do
  code=$(echo $line | awk '{print $1}' FS=\|)
  cn=$(awk -v CID=$code '$1==CID {print $2}' FS=\| id.txt)
  echo $line|awk -v CN=$cn 'BEGIN {FS=OFS="|"} {$1=CN} 1'
done

Awk solutions:

$ awk '
    BEGIN {FS=OFS="|"}
    FNR==NR{a[$1]=$2;next}
    $1 in a{print a[$1],$2,$3,$4}
' id.txt main.txt

Output:
ZM|1241503759|ax|90
HK|1241503760|ay|10
US|1241503823|az|90
IN|1241503947|at|80

or

$ awk '
    BEGIN {FS=OFS="|"}
    FNR==NR{a[$1]=$2;next}
    $1 in a{print a[$1]"("$1")",$2,$3,$4}
' id.txt main.txt

Output:
ZM(512)|1241503759|ax|90
HK(234)|1241503760|ay|10
US(122)|1241503823|az|90
IN(123)|1241503947|at|80

or

$ awk '
    BEGIN {FS=OFS="|"}
    FNR==NR{a[$1]=$2;next}
    $1 in a{print a[$1],$0}
' id.txt main.txt

ZM|512|1241503759|ax|90
HK|234|1241503760|ay|10
US|122|1241503823|az|90
IN|123|1241503947|at|80

Related post:
- Performing join using awk here
- Update file based on another file in awk here
- Difference between awk NR and FNR variables here
- Replace column with column of another file in awk here
- Match words between two files in bash here
- Delete line based on another file here