Saturday, December 20, 2014

Unix - Delete blank lines from file

Here's some ways to remove empty or blank lines from a file in Unix. Simple but definitely useful.
$ grep -v '^$' file.txt
$ grep '.' file.txt
$ sed '/^$/d' file.txt
$ sed -n '/^$/!p' file.txt
$ awk NF file.txt
$ awk '/./' file.txt

In Vi editor, in escape mode type
:g/^$/ d
Related posts:

Saturday, December 13, 2014

Execute script on remote server SSH - Unix

Assume you have a complex bash script named '' (containing good number of loops, Awk statements, multiple lines of code etc) that you need to run against one or more remote servers via SSH. One way would be to copy (i.e. scp) the script to the remote server(s) and then do something like this: e.g.
$ scp -2 root@

$ ssh -2 root@ sh /tmp/
If its multiple machines, then you can use bash for loop like this:
$ for ip in ; do echo "Executing on IP: $ip" ; ssh -2 root@$ip 'sh /tmp/' ; done
Now assume you do not want to copy your script to the remote machine(s) and want to execute from your local machine without copying it. Here's a way:
$ cat | ssh -2 root@ /bin/sh
And if its to be executed on multiple machines, you can use a for loop like this:
$ for ip in ; do echo "Executing on IP: $ip" ; cat | ssh -2 root@$ip /bin/sh ; done
I am sure you liked this trick ! Feel free to comment below. Thanks.

Related posts:

Sunday, April 27, 2014

Unix xargs parallel execution of commands

Xargs has option that allows you to take advantage of multiple cores in your machine. Its -P option which allows xargs to invoke the specified command multiple times in parallel. From XARGS(1) man page:
-P max-procs
   Run up to max-procs processes at a time; the default is 1.  If max-procs is 0, xargs will run as many processes as possible at a time.   Use  the  -n  option
   with -P; otherwise chances are that only one exec will be done.

-n max-args
    Use at most max-args arguments per command line.  Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the  -x  
    option is given, in which case xargs will exit.

    This option is a synonym for -Ireplace-str if replace-str is specified, and for -I{} otherwise.  This option is deprecated; use -I instead.
Let me try to give one example where we can make use of this parallel option avaiable on xargs. e.g. I got these 8 log files (each one is of 1.5G size) for which I have to run a script named which does some calculation around the log lines in the log file.
$ ls -1 *.out
The script takes nearly 20 seconds for a single log file. e.g.
$ time ./ log1.out

real 0m20.509s
user 0m20.967s
sys 0m0.467s
If we have to run for each of the 8 log files one after the other, total time needed:
$ time ls *.out | xargs -i ./ {}           

real 2m45.862s
user 2m48.152s
sys 0m5.358s
Running with 4 parallel processes at a time (I am having a machine which is having 4 CPU cores):
$ time ls *.out | xargs -i -P4 ./ {} 

real 0m44.764s
user 2m55.020s
sys 0m6.224s
We saved time ! Isn't this useful ? You can also use -n1 option instead of the -i option that I am using above. -n1 passes one arg a time to the run comamnd (instead of the xargs default of passing all args).
$ time ls *.out | xargs -n1 -P4 ./

real 0m43.229s
user 2m56.718s
sys 0m6.353s

Related posts:
- UNIX handle kill when no PID available 

© Jadu Saikia