Saturday, March 7, 2015

Unix - run multiple commands on remote machines in parallel

Assume you have a requirement of running two commands on 10 hosts.

In this post I would like to show you how you can:

1) Run multiple commands on remote host using ssh.

2) Also instead of working on one host at a time serially, how can we execute the commands on multiple hosts in parallel to save the overall execution time.

Find the number of files under '/sdb/_read/' and '/sdb/_write/' directories on following 10 hosts.

$ cat host.txt

Serial execution: 
We can write a bash for loop and work on each host like this:

$ for ip in $(cat host.txt); do echo -n "$ip|"; ssh -2 root@$ip 'R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l) ; echo "$R|$D"'; done

Time taken:
real 0m50.084s
user 0m0.052s
sys 0m0.032s

Parallel execution:
I have already blogged about Unix xargs parallel execution option here, using that:

$ cat host.txt  | xargs -n 1 -P 5 -i ssh -2 root@{} 'I=$(hostname -i);R=$(ls /sdb/_read/ | wc -l ) ; D=$(ls /sdb/_write/ | wc -l)  ; echo "$I|$R|$D"'

Time taken:
real 0m10.437s
user 0m0.044s
sys 0m0.036s

10 Seconds vs 50 Seconds, wow !


Note: Utilities like GNU parallel or Parallel SSH are always the best tool in this job.

Related posts:

  1. Unix Xargs paralel execution
  2. Unix - Execute script on remote host using SSH without copying the script.
  3. Diff remote files using SSH in Unix
  4. Bash and SSH some examples

1 comment:

Anonymous said...

May be you can format your post again for the lines of code to be shown. Now, they're being cut.

© Jadu Saikia