• single program -- the identical code runs on multiple workers.
• multiple data -- each worker can have different, unique data for that code.
• numlabs – number of workers.
• labindex -- each worker has a unique identifier between 1 and numlabs.
• point-to-point communications between workers : labsend, labreceive, labsendrecieve
• Ideal for:  1) programs that takes a long time to execute. 2) programs operating on large data set.

spmd
   labdata = load(['datafile_' num2str(labindex) '.ascii'])                 % multiple data
   result = MyFunction(labdata);                                            % single program
end
        

a = 0.13;
b = 1.53;
n = 120000000;
spmd
   aa = (labindex - 1) / numlabs * (b-a) + a;
   bb = labindex / numlabs * (b-a) + a;
   nn = round(n / numlabs)
   q_part = quad_fun(nn, aa, bb);
end
q = sum([q_part{:}]);
        

8.26x
>> tic
quad_fun_spmd
t3=toc

t3 =
0.7006
        

Distributed Array

• Distributed Array - Data distributed from client and access readily on client. Data always distributed along the last dimension, and as evenly as possible along that dimension among the workers.
• Codistributed Array – Data distributed within spmd. Array created directly (locally) on worker.

Examples

Matrix Multiply (A*B) by Distributed Array

parpool('local');
A = rand(3000);
B = rand(3000);
a = distributed(A);
b = distributed(B);
c = a*b;                             % run on workers automatically as long as c is of type distributed
delete(gcp);
        

Matrix Multiply (A*B) by Distributed Array

parpool('local');
A = rand(3000);
B = rand(3000);
spmd
   u = codistributed(A, codistributor1d(1));  % by row
   v = codistributed(B, codistributor1d(2));  % by column
   w = u * v;
end
delete(gcp);
        

Linear solver (Ax = b)

n = 10000;
M = rand(n);
X = ones(n, 1);
A = M + M';
b = A * X;
u = A \ b;
                

>>tic;linearSolver;t1=toc

t1 =
8.4054
                

n = 10000;
M = rand(n);
X = ones(n, 1);
spmd
   m = codistributed(M, codistributor('1d', 2));     % distribute one portion of m to each MATLAB worker 
   x = codistributed(X, codistributor('1d', 1));     % distribute one portion of x to each MATLAB worker
   A = m + m';
   b = A * x;
   utmp = A \ b;
end
u1 = gather(utmp);                                     % gathering u1 from all MATLAB workers to MATLAB client
        

>>parpool('local');
Starting parallel pool(parpool) using the 'local' profile ... connected to 12 workers
>>tic;linearSolverSpmd;t2=toc
t2 =
16.5482
        

Q: why spmd version needs more time for job completion ?

A: It needs extra time for distribute/consolidate large data and tranfer them to/from different workers

• Computation inside the loop is simple.
• Memory limitations. (create more data compare to serial code)
• Transfer data is time consuming
• Unbalanced computational load
• Synchronization

Contrast Enhancement

contract_enhance.m 

function y = contrast_enhance (x)
   x = double (x);
   n = size (x, 1);
   x_average = sum ( sum (x(:, :))) /n /n ;
   s = 3.0;           % the contrast s should be greater than 1
   y = (1.0 – s ) * x_average + s * x((n+1)/2, (n+1)/2);
   return
end
        

% contrat_serial.m
x = imread('surfsup.tif');
yl = nlfilter (x, [3,3], @contrast_enhance);
y = uint8(y);
                

% contrat_parallel.m
x = imread('surfsup.tif');
xd = distributed(x);
spmd
   xl = getlocalPart(xd);
   xl = nlfilter(xl, [3,3], @contrast_enhance);
end
y = [xl{:}];
        

Problem : When the image is divided by columns among the workers, artifcial internal boundaries are created !

Reason : Zero padding at edges when applying sliding-neighborhood operation.

Solution: Build up communication between workers.

contrast_MPI.m

x = imread('surfsup.tif');
xd = distributed(x);

spmd
   xl = getLocalPart ( xd );


   % find out previous & next by labindex
   if ( labindex ~= 1 )
      previous = labindex - 1;
   else
      previous = numlabs;
   end

   if ( labindex ~= numlabs )
      next = labindex + 1;
   else
      next = 1;
   end


  % attach ghost boundaries
  column = labSendReceive ( previous, next, xl(:,1) );
  if ( labindex < numlabs )
     xl = [ xl, column ];
  end

  column = labSendReceive ( next, previous, xl(:,end) );
  if ( 1 < labindex )
     xl = [ column, xl ];
  end

  xl = nlfilter ( xl, [3,3], @contrast_enhance );


  % remove ghost boundaries
  if ( 1 < labindex )
     xl = xl(:,2:end);
  end

  if ( labindex < numlabs )
     xl = xl(:,1:end-1);
  end


   xl = uint8 ( xl );
end
y = [ xl{:} ];
        

12 workers, running time is 2.01 s, Speedup  = 7.23x

pmode allows the interactive parallel execution of MATLAB® commands. pmode achieves this by defining and submitting a communicating job, and opening a Parallel Command Window connected to the workers running the job. The workers then receive commands entered in the Parallel Command Window, process them, and send the command output back to the Parallel Command Window. Variables can be transferred between the MATLAB client and the workers.

Q: How many workers can I use for MATLAB parallel pool ?

Parallel Computing Toolbox enables you to program MATLAB to use your computer's graphics processing unit (GPU) for matrix operations. For some problems, execution in the GPU is faster than in CPU.

To enable GPU computing of MATLAB on BioHPC cluster:

1. reserve a GPU node by remoteGPU or webGPU
2. inside terminal, type in  export CUDA_VISIBLE_DEVICES=“0” to enable the hardware rendering.

Recall that the serial job only needs 5.7881 seconds, MDCS needs more time for Example 1 as it is not designed for small jobs.