Saturday, March 7, 2015

Unix - run multiple commands on remote machines in parallel

Assume you have a requirement of running two commands on 10 hosts.

In this post I would like to show you how you can:

1) Run multiple commands on remote host using ssh.

2) Also instead of working on one host at a time serially, how can we execute the commands on multiple hosts in parallel to save the overall execution time.

Find the number of files under '/sdb/_read/' and '/sdb/_write/' directories on following 10 hosts.

$ cat host.txt

Serial execution: 
We can write a bash for loop and work on each host like this:

$ for ip in $(cat host.txt); do echo -n "$ip|"; ssh -2 root@$ip 'R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l) ; echo "$R|$D"'; done

Time taken:
real 0m50.084s
user 0m0.052s
sys 0m0.032s

Parallel execution:
I have already blogged about Unix xargs parallel execution option here, using that:

$ cat host.txt  | xargs -n 1 -P 5 -i ssh -2 root@{} 'I=$(hostname -i);R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l)  ; echo "$I|$R|$D"'

Time taken:
real 0m10.437s
user 0m0.044s
sys 0m0.036s

10 Seconds vs 50 Seconds, wow !


Note: Utilities like GNU parallel or Parallel SSH are always the best tool in this job.

Related posts:

  1. Unix Xargs paralel execution
  2. Unix - Execute script on remote host using SSH without copying the script.
  3. Diff remote files using SSH in Unix
  4. Bash and SSH some examples

Thursday, January 1, 2015

Happy new year 2015 to all unix lovers

Wishing all the readers of a Happy and Healthy 2015 ahead, keep rocking !

Saturday, December 20, 2014

Unix - Delete blank lines from file

Here's some ways to remove empty or blank lines from a file in Unix. Simple but definitely useful.
$ grep -v '^$' file.txt
$ grep '.' file.txt
$ sed '/^$/d' file.txt
$ sed -n '/^$/!p' file.txt
$ awk NF file.txt
$ awk '/./' file.txt

In Vi editor, in escape mode type
:g/^$/ d
Related posts:

© Jadu Saikia