UNIX Command Line

First file:

$ cat ranks.txt
172.22.22.12:1
172.22.22.13:1
172.22.22.16:2
172.22.22.10:1
172.22.22.18:3


Second file (Exception file)

$ cat except_ip.txt
172.22.22.13
172.22.22.12


Requirement:

- based on the entries in the second file, i.e. except_ip.txt, update the second fileld of first file i.e. ranks.txt in such a way that : if the IP address is present in the second file, make the rank(2nd field of ranks.txt) to 0, rest entries in ranks.txt should be unchanged.

i.e.

required output:

172.22.22.12:0
172.22.22.13:0
172.22.22.16:2
172.22.22.10:1
172.22.22.18:3


The awk code:

$ awk 'BEGIN {FS=OFS=":"}
FNR==NR {ex[$1]; next}
{ $2 = ($1 in ex) ? "0" : $2; print }
' except_ip.txt ranks.txt

172.22.22.12:0
172.22.22.13:0
172.22.22.16:2
172.22.22.10:1
172.22.22.18:3


.

Input file:

$ cat keyval.txt
Geo=Asia
Sub=Awk
Pt=7.8


Output required:


keys=Geo|Sub|Pt val=Asia|Awk|7.8



$ awk -F= 'END { printf "keys=%s val=%s\n", k, v }
{ k = k ? k s $1 : $1; v = v ? v s $2 : $2 }
' s="|" keyval.txt


.

From mysql prompt:


$ /usr/bin/mysql -u root

...
...

mysql> select * from NMS.main;
+------+-------+
| id | data |
+------+-------+
| 1 | slow |
| 3 | pizza |
| 4 | bash |
| 6 | awk |
+------+-------+
4 rows in set (0.00 sec)

mysql> select id,data from NMS.main
-> where id in (3,4);
+------+-------+
| id | data |
+------+-------+
| 3 | pizza |
| 4 | bash |
+------+-------+
2 rows in set (0.00 sec)



Now if we have to automate the same stuffs inside a bash script, this is how we can achieve that.


$ cat execmysql.sh

#!/bin/sh

ids="3,4"
table="NMS.main"

qry="select id,data from $table where id in ($ids)"

echo "Executing the following query"
echo $qry

/usr/bin/mysql -u root << eof
$qry
eof


Executing:


$ ./execmysql.sh
Executing the following query
select id,data from NMS.main where id in (3,4)
id data
3 pizza
4 bash


Input file:


$ cat studentdt.dat
Entry C-3500
Name: MrA
Roll: 34/2K
Age: 24
Spl Paper: AIX
Entry T-3405
Name: MrB
Roll: 32/2K
Age: 25
Spl Paper: Compiler
Entry C-9500
Name: MrsD
Roll: 37/2K
Age: 27
Spl Paper: Arch


Output required:

- Add the "entry name" as prefix to each line which is related to that particular entry(C-3500,T-3405 ... etc)

i.e.


C-3500:Name: MrA
C-3500:Roll: 34/2K
C-3500:Age: 24
C-3500:Spl Paper: AIX
T-3405:Name: MrB
T-3405:Roll: 32/2K
T-3405:Age: 25
T-3405:Spl Paper: Compiler
C-9500:Name: MrsD
C-9500:Roll: 37/2K
C-9500:Age: 27
C-9500:Spl Paper: Arch



$ awk '
BEGIN { OFS=":" }
{
if ($1 == "Entry") {
e=$2
} else {
split($0, d, ":");
print e, d[1],d[2];
}
}
' studentdt.dat


Similar post:


- Sort and add prefix using sed, bash


#The details file, Id,Name,Age


$ cat details.txt
IDxx,MrA,43
IDtt,MrsD,35
IDkk,MrN,67
IDaa,MrC,25
IDdd,MrsQ,78


#location file, which contain location information for some Ids


$ cat location.txt
IDkk,IN
IDxx,GB
IDaa,US


Output required: join both the files details.txt and location.txt
#Id,Name,Location(If present),Age


IDaa,MrC,US,25
IDdd,MrsQ,,78
IDkk,MrN,IN,67
IDtt,MrsD,,35
IDxx,MrA,GB,43



$ sort -t "," -k1 details.txt > details.txt.srt
$ sort -t "," -k1 location.txt > location.txt.srt

$ cat details.txt.srt
IDaa,MrC,25
IDdd,MrsQ,78
IDkk,MrN,67
IDtt,MrsD,35
IDxx,MrA,43

$ cat location.txt.srt
IDaa,US
IDkk,IN
IDxx,GB


A good learn from unix.com

join(1) is a relational database operator. If your version of join conforms to POSIX.1:2004 or SUSv3 i.e.supports the -0 0 outer join operation, the following

$ join -t, -a 1 -a 2 -o 0,1.2,2.2,1.3 details.txt.srt location.txt.srt

Related Post:

bash join command
join using awk

Suppose:


var="kde24hx0ep1024x960rstp390"


Now if we have to extract the digit part before x (i.e. 1024) and digit part after x (i.e. 960), here is a perl one liner for the same.

# perl -e '$var="kde24hx0ep1024x960rstp390"; $var=~/(\d{1,})x(\d{1,})/; print $1."\t".$2."\n";'
1024 960

or

# perl -e '$var="kde24hx0ep1024x960rstp390"; ($pos1,$pos2)=($var=~/(\d+)x(\d+)/); print $pos1."\t".$pos2;'
1024 960

Count particular character in a string - bash

Suppose:


$ string="unix bash script tutorial"


Now if I have to count the number of "i" s in the $string,


$ string="${string//[^i]/}"; echo ${#string}
3

or

$ echo $((`echo $string|sed 's/[^i]//g'|wc -m`-1))
3


Where
-m (print the character counts, an wc option)

This is how we can combine or concatenate arrays in bash scripting.


$ cat combinearr.sh
#!/bin/sh
#Combine or concatenating arrays in Bash
# Source : http://tldp.org/LDP/abs/html/arrays.html

# Subscript packed.
declare -a arrA=( A1 B1 C1 )

echo "____________"
echo "Array arrA"
echo "____________"
for (( i = 0 ; i < 3 ; i++ ))
do
echo "Element [$i]: ${arrA[$i]}"
done

# Subscript sparse ([1] is not defined).
declare -a arrB=( [0]=A2 [2]=C2 [3]=D2 )

echo "____________"
echo "Array arrB"
echo "____________"
for (( i = 0 ; i < 4 ; i++ ))
do
echo "Element [$i]: ${arrB[$i]}"
done

declare -a arrayX

#Combine arrA and arrB
arrayX=( ${arrA[@]} ${arrB[@]} )

echo "____________"
echo "Array arrayX"
echo "____________"
echo "${arrayX[@]}"

cnt=${#arrayX[@]}

echo "____________"
echo "Array arrayX"
echo "____________"
for (( i = 0 ; i < cnt ; i++ ))
do
echo "Element [$i]: ${arrayX[$i]}"
done


Running the script:

$ ./combinearr.sh
____________
Array arrA
____________
Element [0]: A1
Element [1]: B1
Element [2]: C1
____________
Array arrB
____________
Element [0]: A2
Element [1]:
Element [2]: C2
Element [3]: D2
____________
Array arrayX
____________
A1 B1 C1 A2 C2 D2
____________
Array arrayX
____________
Element [0]: A1
Element [1]: B1
Element [2]: C1
Element [3]: A2
Element [4]: C2
Element [5]: D2


More of arrays in Bash can be found here

I learn this very useful trick from http://www.zazzybob.com/bin/calc.html


$ cat mybc
#!/bin/sh

( echo scale=2; echo "$@" ) | bc

exit 0


scale=The value of the scale function is the number of digits after the decimal point in the expression.

Now normal calculations became very simple with bc (make mybc an alias for bc or use separately):


$ mybc 34256+123456
157712

$ mybc 123456-34256
89200

$ mybc 123456*34256
4229108736

$ mybc 123456/34256
3.60

$ mybc 123456+34256*2
191968


An example to illustrate the use of arrays in bash scripting.


#!/bin/sh
#Bash array implementation

array=(bash ksh csh)
len=${#array[*]} #Num elements in array

echo "Array has $len members.They are:"

i=0
while [ $i -lt $len ]; do
echo "$i: ${array[$i]}"
let i++
done

#Some operations
echo "Adding \"zsh\" to the end of the array"
array=( "${array[@]}" "zsh" )
echo "Listng all the elements in the array"
echo "${array[@]}"
echo "Listng all the elements in the array"
echo "${array[*]}" #One more way
echo "Deleting \"ksh\""
unset array[1] #Same as array[1]=
echo "Now the array is"
echo "${array[@]}"
echo "length of 3rd element in the array"
len1=${#array[2]}; echo $len
echo "Deleting the whole array"
unset array
echo "${array[@]}" #Array is empty


Executing:


$ ./basharray.sh
Array has 3 members.They are:
0: bash
1: ksh
2: csh
Adding "zsh" to the end of the array
Listng all the elements in the array
bash ksh csh zsh
Listng all the elements in the array
bash ksh csh zsh
Deleting "ksh"
Now the array is
bash csh zsh
length of 3rd element in the array
3
Deleting the whole array



A null/empty array in bash can be declared as (array initialization)


array=()


So you came to know the followings:

- How to initialize an array
- How to add an element to an array
- How to delete a particular element in the array or to delete the whole array.
- How to find the length of an element in an array or length of the whole array.

A more about Array of BASH can be found here:
http://tldp.org/LDP/abs/html/arrays.html

Lets have a script to compare some float values.


$ cat floatcomp.sh
#!/bin/sh

array=(0.255 0.8569 5.356 3.8521)
max=0
len=${#array[*]}
i=0

while [ $i -lt $len ]
do
echo "$i: ${array[$i]}"
val=`echo "${array[$i]}" `
if [ $val -gt $max ]
then
max=$val
fi
let i++
done
echo "MAX IS => $max"


Running the script:

0: 0.255
./floatcomp.sh: line 12: [: 0.255: integer expression expected
1: 0.8569
./floatcomp.sh: line 12: [: 0.8569: integer expression expected
2: 5.356
./floatcomp.sh: line 12: [: 5.356: integer expression expected
3: 3.8521
./floatcomp.sh: line 12: [: 3.8521: integer expression expected
MAX IS => 0

HUH!! Where is the problem ? Got it, problem is with the if statement above we are using to compare two float values, we just can't compare float values like the ints and strings.
To make it work, we have to use "bc" command. Some of its sample uses are below:


$ echo "3.2 > 3.2" | bc
0

$ echo "3.2 > 3.4" | bc
0

$ echo "3.2 < 3.4" | bc
1


Note: Notice the output of the above comparisons.

Based upon it, we will modify our script, the if condition will be like this


if [ $(echo "$max < $val"|bc) -eq 1 ]

instead of

if [ $val -gt $max ]


So the script will look like this:

$ cat floatcomp1.sh
#!/bin/sh

array=(0.255 0.8569 5.356 3.8521)
max=0
len=${#array[*]}
i=0

while [ $i -lt $len ]
do
echo "$i: ${array[$i]}"
val=`echo "${array[$i]}" `
if [ $(echo "$max < $val"|bc) -eq 1 ]
then
max=$val
fi
let i++
done
echo "MAX IS => $max"


The output:

$ ./floatcomp1.sh
0: 0.255
1: 0.8569
2: 5.356
3: 3.8521
MAX IS => 5.356


As "expr" does not help in float computations, here are some of the alternatives

___________________________
float computations using bc
___________________________

$ Number=`echo 80 \* 10.69 | bc`; echo $Number

855.20

_____________________________
float computations using awk
_____________________________

$ Number=`(echo - | awk '{ print 80*10.69}')`; echo $Number

855.2

And if you are supplying something from outside

$ Number=`(echo - | awk '{ print 80*10.69}')`; echo $Number

855.2

One more way of passing variables to AWK

$ S=80; Number=`echo - | awk -v K=$S '{ print K*10.69}'` ; echo $Number

855.2

_____________________________________
Search a letter in a string using grep
_____________________________________

$ string="unix-bash 2389"

$ character="-"

$ echo $string | grep -q "$character" && echo "found" || echo "not found"
found

$ character="@"

$ echo $string | grep -q "$character" && echo "found" || echo "not found"
not found

_______________________________________
Search a letter in a string using awk
_______________________________________

$ character="@"

$ echo $string | awk -vc="$character" '{if(gsub(c,"")) print "Found";else print "Not Found"}'
Not Found

$ character="-"

$ echo $string | awk -vc="$character" '{if(gsub(c,"")) print "Found";else print "Not Found"}'
Found

wait command stop script execution until all jobs running in background have terminated, or until the job number or process id specified as an option terminates.
It returns the exit status of waited-for command.

wait can take the job-id or the process number. i.e.

wait%1 or wait $PID

_________
wait ${!}
_________

wait ${!} means "to wait till the last background process is completed" ($! being the PID of the last background process)

________________________
An example on wait command.
________________________

Suppose
- you have a script called sort_db.sh which sorts some data files and takes a lot of time to complete(you definitely want it to run as background process in your script)
- One more script called bkptmp.sh, which does some job of backing up some tmp files(nowhere related to the above sort_db.sh)
- You have to perform some tasks in your script after sort_db.sh and bkptmp.sh complete their individual tasks (Note: both the .sh should be completed before you perform the said operation)


$ cat waittest.sh
#!/bin/sh

./sort_db.sh &
echo "1st Line"

./bkptmp.sh &
echo "2dn line"

wait
echo "Some operation will follow this"
...
...


In such situations the "bash wait command" is useful. It will wait till sort_db.sh and bkptmp.sh get complete their execution.

You might be thinking, we could have run sort_db.sh and bkptmp.sh in foreground, so that the execution of the operation will follow them. The problem is that you don't want to wait for sort_db.sh to complete for bkptmp.sh to start (I told earlier, they are not dependent). So using wait, the time sort_db.sh gets completed, we will be done(or almost done) with bkptmp.sh.

Input filea is # delimited.


$ cat filea
a#b#c#0
d#e#f#1
g#h#i#1


Purpose:
Remove the 2nd and 3rd column from filea


$ awk 'BEGIN{FS=OFS="#"}{$2=$3=""}{print}' filea

a###0
d###1
g###1


$ awk 'BEGIN{FS=OFS="#"}{$2=$3="";gsub(FS"+",FS)}1' filea

a#0
d#1
g#1

As I told earlier , the 1 in the above code is similar to {print $0}.

Similar Post:
- Replace the 100's in 3rd field with different numbers

This is how we can create a menu (continuous) in bash scripting.

If you need to write a small script to run your "tests suite", I am presenting here an efficient way(I feel so) of writing a menu.

_________
Way 1
_________


#!/bin/sh

f_Reg () {
echo "Regression function"
echo "All your logic in this function"
}

f_Smoke () {
echo "Smoke function"
echo "All your logic in this function"
}

while : # Loop forever
do
cat << !

R U N M E N U

1. Regression Tests
2. Smoke Tests
3. Quit

!

echo -n " Your choice? : "
read choice

case $choice in
1) f_Reg ;;
2) f_Smoke ;;
3) exit ;;
*) echo "\"$choice\" is not valid "; sleep 2 ;;
esac
done


Executing:


$ ./menu.sh

R U N M E N U

1. Regression Tests
2. Smoke Tests
3. Quit

Your choice? :


________
Way2:
________

The other way of making a continuous menu is to have a variable set to "no", and exit the menu when the value is "yes" (set the value to "yes" for the quit option in the menu like below):


#!/bin/sh

quit="no"

f_Reg () {
echo "Regression function"
echo "All your logic in this function"
}

f_Smoke () {
echo "Smoke function"
echo "All your logic in this function"
}

while [ $quit != "yes" ]
do
echo "1. Regression Tests"
echo "2. Smoke Tests"
echo "3. Quit"
echo -n "Your choice? : "
read choice

case $choice in
1) f_Reg ;;
2) f_Smoke ;;
3) quit="yes" ;;
*) echo "\"$choice\" is not valid"
sleep 2 ;;
esac
done


Executing:


$ ./menu1.sh
1. Regression Tests
2. Smoke Tests
3. Quit
Your choice? :


I have a huge set of(more than 300) log files(nmn_log.*.txt) in one of my logdir. I had to find out the latest log file name.

$ ls -lrt
total 475244
-rw-r--r-- 1 jks staff 13464 May 20 06:34 nmn_log.23219.txt
-rw-r--r-- 1 jks staff 96851 May 20 06:38 nmn_log.23236.txt
...
...
-rw-r--r-- 1 jks staff 96969 Jun 4 11:59 nmn_log.23233.txt
-rw-r--r-- 1 jks staff 91016 Jun 4 12:03 nmn_log.23239.txt
drwxr-xr-x 2 jks staff 4096 Jun 4 12:04 old

The intention is to print the last field of the last line (except the dir old) from the "ls -lrt" output in that directory.


$ ls -lrt | awk '/nmn_log/ { f=$NF };END{ print f }'
nmn_log.23239.txt


So basically the general way of printing the last field of last line(in this case the last field of last line of "ls -lrt" command):


$ ls -lrt | awk '{ f=$NF }; END{ print f }'


Input File:

$ cat file.txt
MrA|1|2|3
MrB|4|5|6
MrC|7|8|9

Suppose I have to increment last two fields in file.txt by 10. i.e. required output:

MrA|1|12|13
MrB|4|15|16
MrC|7|18|19

The basic solution using awk:


$ awk '
BEGIN{OFS=FS="|"; inc=10}
{$NF=$NF+inc
$(NF-1)=$(NF-1)+inc} {print}
' file.txt


Now using awk function:

Making it more generic with a function f_inc. The function takes two args
1) Number of last fields to be incremented by 2)"inc" value


$ awk '
BEGIN { OFS=FS="|" }
function f_inc(f,inc) {
for (i=0; i<f; i++) {
$(NF-i)=$(NF-i)+inc;
}
}
{
f_inc(2,10);
print;
}
' file.txt


Output:

MrA|1|12|13
MrB|4|15|16
MrC|7|18|19