[edit] Using MATLAB on the ITS HPC Cluster

MATLAB is a software package for general mathematical work. MATLAB is installed for both serial (single-processor) and distributed (multi-processor) use. Version R2007b of MATLAB is installed on the cluster.

[edit] Single-processor interactive use

Single-processor interactive MATLAB jobs can be run on the master node for testing, debugging, and short production runs. Otherwise, MATLAB jobs should be run via submission to the PBS batch queue using the "qsub" command. For single-processor interactive MATLAB jobs, you must modify your environment using the shell command

   module load matlab

Once the module is loaded, MATLAB can be started by giving the shell command

   matlab

[edit] Single-processor batch use

Single-processor MATLAB batch jobs require that you write a PBS batch script and submit the script to the PBS batch queue for execution. Within the batch script, you need to load the "matlab" module with the command

   module load matlab

so that the PATH and other environment variables are set properly in the batch job to run MATLAB.

Following is a PBS script that can be used to run a serial MATLAB batch job on the cluster. The sample script includes a request for one processor and 10 hours of wall time. Note that the script includes the command to load the MATLAB worker module as mentioned above and specifies the "-nodisplay" option. The script specifies that the actual MATLAB program (or "M-file") to be run is contained in the file "MyJob.m".

  #PBS -N matlab_test
  #PBS -l walltime=10:00:00
  #PBS -l nodes=1:ppn=1
  #PBS -j oe
  #
  # Load the MATLAB module
  module load matlab
  #
  # cd to the directory where the job was submitted
  cd $PBS_O_WORKDIR
  #
  # Run MATLAB
  matlab -nodisplay -r MyJob

This script is submitted to the batch queue using the command

  qsub matlab_test.pbs

where matlab_test.pbs is the name of the file containing the script.

The PBS batch script given above, with appropriate minor modifications, can be used to submit a wide variety of MATLAB jobs. For example, if the following sample MATLAB M-file (taken from the Western Canada Research Grid) is placed in a file named input.m, you can run the job using this script. The Postscript output will be put into the file matlab_test_plot.ps. You can send this file to a Postscript printer or view it on-screen using a Postscript viewer like Ghostscript.

  % MATLAB M-file example to approximate a sawtooth
  % with a truncated Fourier expansion.
  %
  nterms=5;
  fourbypi=4.0/pi;
  np=100;
  y(1:np)=pi/2.0;
  x(1:np)=linspace(-2.0*pi,2*pi,np);
  for k=1:nterms
     twokm=2*k-1;
     y=y-fourbypi*cos(twokm*x)/twokm^2;
  end;
  plot(x,y);
  print -deps matlab_test_plot.ps;
  quit;

[edit] Multi-Processor Distributed Batch Use - correctness not yet verified

Multi-processor, distributed MATLAB makes use of the MATLAB Distributed Computing Engine (MDCE) on the cluster. The use of the combination of MDCE and PBS causes the process of running distributed MATLAB jobs to be a bit more complex than in the serial case discussed previously.

An overview of how distributed MATLAB jobs run using MDCE and PBS is helpful. In the distributed case, the user usually writes the MATLAB program as an M-file that delegates computational tasks to other "worker" M-files and collects the results they generate for possible additional processing. The user writes a PBS script that contains the MATLAB command to run the first M-file, and then submits the script to the PBS queue. When this job is run, MATLAB builds additional PBS scripts and submits them to the queue in order to request additional compute nodes. These additional nodes run the "worker" M-files in MATLAB kernels executing in those nodes. These instances of MATLAB are known as "workers". Typically, the first M-file waits for all the workers to complete their processing in order to obtain the results they generate. Then the first M-file may carry out additional processing tasks using those results and produce output according to the needs of the user.

[edit] Example

A more detailed description follows and is driven by an example. In this example, each of five MATLAB workers will be used to create a matrix of randomly-generated numbers and return this matrix to the top-level MATLAB instance for possible further processing. Each worker will run the following M-file "myTask.m" to create its matrix:

  function m = myTask(i)
  rand('state', sum(100*clock));
  m = rand(i,i);
  end

In order to cause each of five workers to run this M-file, the user will submit a job to the scheduler using the following PBS batch script

  #PBS -N myJob
  #PBS -l walltime=10:00:00
  #PBS -l nodes=1:ppn=1
  #PBS -j oe
  #
  # Load the MATLAB module
  module load matlab
  #
  # cd to the directory where the job was submitted
  cd $PBS_O_WORKDIR
  #
  # Run MATLAB
  matlab -nodisplay -r myJob

This script just asks for a single processor on which to run a serial job named "myJob" that exists as an M-file "myJob.m". Note that the environment is set for a serial MATLAB job.

The M-file myJob.m that is run by this script looks like this:

  sched = findResource('scheduler', 'type', 'generic');
  set(sched, 'DataLocation','/home/rab5/myProjectDirectory');
  set(sched, 'HasSharedFilesystem', true);
  set(sched, 'SubmitFcn', 'submitFunc');
  j = createJob(sched);
  
  for i = 1:5
      createTask(j, @myTask, 1, {i})
  end
  
  get(sched)
  submit(j)
  waitForState(j)
  for i=1:length(j.Tasks)
      j.Tasks(i).OutputArguments{1}
  end
  
  destroy(j)

The bold parts of myJob.m are specific to this example and should be modified as needed to run a different job. Note that this M-file creates five tasks, all copies of myTask.m with different parameter values, and submits those tasks to the PBS queue using the "submit" function.

[edit] Submit Function

A submit function is a MATLAB function that constructs a PBS batch script and submits it to the PBS queue for execution on the cluster. In the example above, the M-file myJob.m calls a submit function to submit PBS requests for cluster resources on which the workers will run. The submit function, in this case named "submitFunc", must be customized for the job to be run. For this example, the submit function is

 function submitFunc(scheduler, job, props, varargin)
 prevDir = cd('/home/rab5/matlabtest');
 for i = 1:props.NumberOfTasks
    filename = sprintf('/home/rab5/matlabtest/Task%d', i);
    fid = fopen(filename, 'w');
    fprintf(fid, '#!/bin/sh\n');
    fprintf(fid, '#\n');
    fprintf(fid, '#PBS -N myTask\n');
    fprintf(fid, '#PBS -l walltime=2:00\n');
    fprintf(fid, '#PBS -l nodes=1:ppn=1:ethernet\n');
    fprintf(fid, '#PBS -j oe\n');
    fprintf(fid, '#PBS -v MDCE_DECODE_FUNCTION=decodeFunction\n');
    fprintf(fid, '#PBS -v MDCE_STORAGE_LOCATION=%s\n', props.StorageLocation);
    fprintf(fid, '#PBS -v MDCE_STORAGE_CONSTRUCTOR=%s\n', props.StorageConstructor);
    fprintf(fid, '#PBS -v MDCE_JOB_LOCATION=%s\n', props.JobLocation);
    fprintf(fid, '#PBS -v MDCE_TASK_LOCATION=%s\n', props.TaskLocations{i});
    fprintf(fid, 'export MDCE_DECODE_FUNCTION=decodeFunction\n');
    fprintf(fid, 'export MDCE_STORAGE_LOCATION=%s\n', props.StorageLocation);
    fprintf(fid, 'export MDCE_STORAGE_CONSTRUCTOR=%s\n', props.StorageConstructor);
    fprintf(fid, 'export MDCE_JOB_LOCATION=%s\n', props.JobLocation);
    fprintf(fid, 'export MDCE_TASK_LOCATION=%s\n', props.TaskLocations{i});
    fprintf(fid, 'cd /home/rab5/matlabtest\n');
    fprintf(fid, 'module load matlab_worker\n');
    fprintf(fid, '%s\n', scheduler.MatlabCommandToRun);
    fclose(fid);
    logLocation = [scheduler.DataLocation filesep props.TaskLocations{i} '.log'];
    system(['date >> ' logLocation]);
    command = ['qsub ' filename ' >> ' logLocation ' &'];
    system(['echo "' command '" >> ' logLocation]);
    system(command);
 end
 cd(prevDir);

This submit function can serve as a starting point for creating submit functions for use with other jobs. The bold portions indicate where the function should be modified for use in running other jobs. Note that the PBS script that is built uses the "matlab_worker" module - a module that sets the environment appropriately for a worker MATLAB kernel running in a compute node that is invoked using MDCE. Advanced users may want to make additional changes according to their needs.

[edit] Using Other Versions of MATLAB

In the description above, the module names "matlab" and "matlab_worker" are actually synonyms for "matlab_R2007b" and "matlab_R2007b_worker". There may be module files for older versions of MATLAB such as R2007a. The presence of module files for an older version of MATLAB suggests, but does not guarantee, that the corresponding older version is still installed and usable. These older versions are unsupported but may be useful for running old scripts that would required updating to run using the current version of MATLAB.

Version R2008a of MATLAB was installed in April 2008. Its modules are named "matlab_R2008a" and "matlab_R2008a_worker".

Generally, two versions of MATLAB are released each year, an "a" version in the spring and a "b" version in the fall. We try to install new versions shortly after they are released.

[edit] MATLAB PARALLEL COMPUTING

Using the parallel computing on the cluster and matlab has been simplified by converting a simple for-loop so that it runs in parallel. An example would be to plot a sin wave and plot the wave form , a simple for-loop would run its iterations like this:

clear A;
for i = 1:1024
   A(i) = sin (i*2*pi/1024);
end
plot(A);

To interactively run code that contains a parallel loop, you first open a MATLAB® pool. This reserves a collection of MATLAB worker sessions to run your loop iterations. The MATLAB pool can consist of MATLAB sessions running on your local machine or on a remote cluster:

matlabpool open;

With the MATLAB pool reserved, you can modify your code to run your loop in parallel by using a parfor statement:

clear A;
parfor i=1:1024
A(i) = sin(i*2*pi/1024);
end
plot(A);

The Only difference in this loop is the keyword parfor instead of for. the results are always the same but the difference is the parfor runs a single iteration on different workers

Because the iterations run in parallel in other MATLAB sessions, each iteration must be completely independent of all other iterations. The worker calculating the value for A(100) might not be the same worker calculating A(500). There is no guaranty of sequence, so A(900) might be calculated before A(400). (The MATLAB Editor can help identify some problems with parfor code that might not contain independent iterations.) The only place where the values of all the elements of the array A are available is in the MATLAB client, after the data returns from the MATLAB workers and the loop completes

Finally, you should release the workers by using the

Matlabpool close;

[edit] TESTING ON THE CLUSTER

To submit this to the cluster use the modified code into a file called test.m

function test
clear A;
tic
parfor (i=1:1024)
  A(i) = sin(i*2*pi/1024);
end
toc
plot(A);
print -deps matlab_test_plot.ps;

and the pbsscipt matlab_test.pbs

 #PBS -N myJob
 #PBS -l walltime=02:00:00
 #PBS -l nodes=1:ppn=1
 #PBS -j oe
 #
 # Load the MATLAB module
 module load matlab
 #
 # cd to the directory where the job was submitted
 cd $PBS_O_WORKDIR
 #
 # Run MATLAB
 matlab -nodisplay -r test

A plot on matlab_test_plot.ps would be created and can be viewed using a Postscript viewer like Ghostscript.

[edit] MONTE CARLO SIMULATION EXAMPLE

Another example that is slightly more complicated is the monte carlo simulation for flipping coins.

function fixedmonteCarlo
%  This example involves running a 
%  monte-carlo simulation that produces 
%  an integer from 1 to 20.  We want to
%  display a histogram of the result.
   N = 500000;
   nCoins = 20;
   numHeads = zeros(1, N);
 tic
 parfor (simNum = 1:N)
   numHeads(simNum) = flipCoins(nCoins);
 end
   changeHeads=sign(diff(numHeads));
 toc
 x=(0:20);
 figure, hist(numHeads,x);
 figure, hist(changeHeads,[-1,0,1])
 print -deps matlab_test_plot.ps;
 
 function counter = flipCoins(nCoins)
 % Simulate the flipping of 20 coins and count the number of heads 
 counter = sum(rand(1, nCoins) > .5);

enter this code into an fixedmonteCarlo.m file and use a pbs script montecarlotest.pbs

#PBS -N myJob
 #PBS -l walltime=10:00:00
 #PBS -l nodes=1:ppn=1
 #PBS -j oe
 #
 # Load the MATLAB module
 module load matlab
 #
 # cd to the directory where the job was submitted
 cd $PBS_O_WORKDIR
 #
 # Run MATLAB
 matlab -nodisplay -r fixedmonteCarlo

using the qsub montecarlotest.pbs to submit the job on the cluster.

A plot on matlab_test_plot.ps would be created and can be viewed using a Postscript viewer like Ghostscript.

[edit] FOR MORE ON PARALLEL COMPUTING

Users Guide Site

Users Guide (PDF)

--Mahmoud.Audu 09:55, June 30, 2008 (EDT)

This article is a stub. You can help by adding to it.