ITS/High Performance Computing Cluster/FAQ
[edit] ITS Cluster Basics
[edit] How many compute nodes are there on the cluster, how fast are they, how much memory do they have, and how are they interconnected?
The cluster has 112 compute nodes. Each of the original 18 nodes has two Intel Pentium 4 Xeon EM64T processors running at 3.2 GHz, 4 Gbytes of main memory, and a 73 Gbyte SCSI hard drive. In December 2006, an additional 24 nodes were installed. Each of these has two Intel Pentium 4 Xeon EM64T processors running at 3.8 GHz, 4 Gbytes of main memory, and a 73 Gbyte SCSI hard drive. In November 2007, an additional 51 nodes were installed: there are 44 nodes each with two Intel Pentium 4 Xeon EM64T dual-core processors running at 3.0 GHz, 8 Gbytes of main memory, and a 73 Gbyte SAS hard drive, and there are 7 nodes each with two Intel Pentium 4 Xeon EM64T quad-core processors running at 2.33 GHz, 12 Gbytes of main memory, and a 146 Gbyte SAS hard drive. In addition, there is an SMP node with four dual-core Intel Pentium 4 Xeon EM64T processors running at 3.0 GHz, with 32 Gbytes of main memory and mirrored 146 Gbyte SCSI hard drives that has not yet been put into production. In June 2008, an additional 18 nodes were installed, each with two Intel Pentium 4 Xeon EM64T quad-core processors running at 3.0 GHz, 16 Gbytes of main memory, and a 146 Gbyte SAS hard drive. All nodes are interconnected with a Gigabit Ethernet network for message passing. In addition to the compute nodes, there is a login or "master" node (hpcc-master.case.edu) used for login, compiling, testing, and submitting batch jobs. The master node has approximately 4 Tbytes of directly attached disk storage. There is also a management node (hpcc1-mgmtnode.case.edu) used for cluster management purposes and to hold documentation. Finally, all nodes have been interconnected by a separate Ethernet for the purpose of out-of-band cluster management.
As of July 2008, 104 compute nodes are in production: eight of the 3.0 GHz quad-core nodes are temporarily reserved for testing and development.
[edit] What operating system runs on the cluster?
Red Hat Enterprise Linux runs on all of the cluster nodes.
[edit] Who is eligible to use the cluster?
Only Case faculty and their designates are eligible to use the cluster.
[edit] How do I get an account on the cluster?
The account application form should be filled out by a Case faculty member and sent to its-cluster-admin@case.edu. Each new cluster account will be set up with a username to match the Case network ID of the account owner, and will be assigned an initial password which should be changed immediately. Until cluster account information is merged into the campus account information, passwords for the cluster will have to be maintained separately from Case network ID passwords.
[edit] Getting Started Using The HPC Cluster
[edit] How do I login to the cluster?
To login to the cluster, establish a secure shell connection to the login / master node hpcc-master.case.edu using an SSH client from a computer with a Case network IP address. Unsecure telnet will not work. Connections from off-campus computers and wireless connections will work only if a Case IP address is obtained via VPN. Once you are connected, type your cluster username in response to the "login as:" prompt, and type your cluster password in response to the "Password:" prompt. Press the "Enter" or "Return" key after each entry.
[edit] How do I change my cluster password?
Once you are logged in to the cluster, type the command
passwd
and respond to the prompts by entering your old password and your new password.
[edit] How do I transfer files to and from the cluster?
An easy way to transfer files to the cluster is by using a secure FTP client on the computer that contains the files to be transferred. Use the secure FTP client to login to the cluster using your cluster username and cluster password. Then use FTP commands such as "put" to transfer the files. To transfer files from the cluster, use a secure FTP client on the computer where you want to transfer the files, login to the cluster as described above, and use an FTP command such as "get" to transfer the files. See the documentation for your FTP client for more details.
[edit] Where do I get the SSH and secure FTP client software?
Windows and Linux/UNIX versions of both clients can be downloaded for free from http://www.putty.nl. There is another popular Windows version available for free at http://www.winscp.net.
[edit] Where do I get the VPN client software?
VPN client software from Cisco is available at http://vpnsetup.case.edu. Alternatively, there is a VPN client built into Windows XP. The VPN article has more information on using VPN at Case.
[edit] Is file sharing among members of a research group possible on the cluster?
Yes. The login node of the cluster runs a version of the Linux operating system and the usual Linux owner/group/world file protection scheme can be used to protect and/or share files. User accounts for members of a research group are created within a single Linux group to facilitate file sharing. It is also possible for individuals to be members of multiple groups.
[edit] Editing And Compiling On The Cluster
[edit] Where can I get some help on using the command line interface on the cluster?
The command line interface is provided by a Linux shell. Several different Linux shells are available for users. By default, users are provided with bash. Other available shells include ksh, csh, and tcsh. To the casual user, all of these shells work very similarly. Information on using the Linux shells is available at http://www.linuxplanet.com/linuxplanet/tutorials/827/1 and http://www.linuxplanet.com/linuxplanet/tutorials/936/1. You can find other shell tutorials by doing a web search.
[edit] What text editors are available on the cluster?
The primary text editor used on UNIX and Linux systems is vi. A tutorial on using vi is available at http://www.unix-manuals.com/tutorials/vi/vi-in-10-1.html. You can find other vi tutorials and information by doing a web search. The emacs editor is also available. There is a tutorial for emacs available at http://www.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html.
[edit] What programming languages are available on the cluster?
Compilers for FORTRAN, C, and C++ are available. For each language, the Intel, GNU, and Portland Group compilers are available. For more information, see the documentation at https://hpcc1-mgmtnode.case.edu and, especially, the section on “Compilers and Libraries” at ITS_High_Performance_Computing_Cluster. Additional programming/scripting languages available on the cluster include the as and as86 assemblers, awk, perl, and python. See their respective man pages on the cluster for more information.
[edit] What programming libraries are available on the cluster?
Please see the section on “Compilers and Libraries” at ITS_High_Performance_Computing_Cluster for current information.
[edit] How do I compile (and run) source code that uses an MPI library?
In order to compile for the Gigabit Ethernet interconnect, you need to ensure that the "mpich" module is loaded before compiling. In addition, you need to ensure that the appropriate module is loaded for the desired version of the compiler you are using. Compilation commands are as follows:
mpicc - Intel C compiler mpiCC - Intel C++ compiler mpif77 - Intel FORTRAN 77 compiler mpif90 - Intel FORTRAN 90 compiler
Using these commands ensures that the required libraries and environment variables are set as appropriate for the interconnect. The "mpich" module and the correct compiler module should also be loaded in the batch script that is submitted to PBS. A sample batch script might look like
#PBS -N MPICH_test #PBS -l walltime=10:00 #PBS -l nodes=2:ppn=2 #PBS -j oe module load mpich cd $PBS_O_WORKDIR mpiexec /home/xyz123/test/hello
Loading the appropriate modules ensures that the proper version of "mpiexec" is used when running the program.
[edit] What are modules?
Simply stated, a module is a packaging of definitions of environment variables into a script. In general, there is a module defined for each application, and that module defines the environment appropriately for that application.
To load the module xxxx, use the command
module load xxxx
To unload it, use
module unload xxxx
To see which modules you have loaded, use
module list
To see what modules are available on the system, use
module avail
To see the contents of the module xxxx, use
module display xxxx
[edit] Running Jobs On The ITS HPC Cluster
[edit] How do I run a job on the cluster?
Simply typing the name of an executable will cause that executable to be run on the master node of the cluster. This is appropriate only for interactive work such as editing and compiling, and for testing and debugging sessions. In order to run a program on processors in one or more of the compute nodes, and enjoy exclusive use of those processors, you need to submit your program as a job to the PBS batch queue. To do this, type
qsub <scriptfile>
where <scriptfile> is the name of the PBS script that describes your job. For more information on using the qsub command, type "man qsub" on the master node.
[edit] What commands can be put into a PBS script?
In general, a PBS script contains commands to specify options to PBS that pertain to the job, file manipulations that are required as a part of the job, and execution of one or more programs that constitute the job. For example, the following script names the job "test123", requests 30 minutes of "wall time", requests one processor in one node for processing, and specifies that stderr and stdout be redirected to the files test123.err and test123.out, respectively. These PBS options are followed by three command lines to be executed by a shell when the job is run on a compute node.
#PBS -N test123 #PBS -l walltime=00:30:00 #PBS -l nodes=1:ppn=1 #PBS -e test123.err -o test123.out cd ~/test ./test123 cp error.txt error.txt max1.txt max2.txt ~/results
This script begins running with a working directory that is the same as the user's home directory. However, the output files test123.err and test123.out will be placed in the user's working directory at the time that the qsub command is given to submit the job to the PBS system. Then the working directory is changed to ~/test. The program test123 is expected to be found in this directory and is executed with ~/test as the working directory. Finally, when the program test123 terminates, the files error.txt, error.txt, max1.txt, and max2.txt are copied from this directory to the directory ~/results.
Several environment variables are available to the script, including
PBS_O_HOST - the name of the host where the qsub command was issued PBS_O_WORKDIR - the working directory where the qsub command was issued PBS_JOBID - the job identifier assigned to the job by the batch system PBS_JOBNAME - the job name supplied by the user
Please see http://www.osc.edu/hpc/training/batch/batch_0404.pdf for a good description of using PBS at the Ohio Supercomputer Center. Within this presentation, the information on their Pentium 4 cluster also pertains to the Case HPC cluster.
[edit] How do I know the status of my job after I've submitted it to the queue?
On the master node, type
qstat -a
to get a status report on all jobs that have been submitted to PBS but not yet completed. A similar report can be obtained using the command "showq". To get a worst-case estimate of when your queued job will begin to run, type
showstart NNNN
where NNNN is the job number. It is also possible to request email notification of job status from within the PBS script. For example,
#PBS -m b
requests email notification when the job begins and
#PBS -m e
requests email notification when the job completes. Note that these notifications are sent to your email account on the cluster. If you want the email notification to be sent to another email address, insert the command
#PBS -M name@somewhere.net
in your PBS script and replace "name@somewhere.net" with that desired email address.
[edit] How do I delete a job I've submitted to the queue? How do I kill one of my jobs that is already running?
To delete a job from the queue, or to kill a job that is already running, use the following command on the master node:
qdel NNNN
where NNNN is the job number.
[edit] How do I request specific nodes or processors for my job?
It is possible to request nodes based on their processor speeds and other characteristics. For example,
#PBS -l nodes=4:single3800
requests 4 nodes with 3.8 GHz single-core processors,
#PBS -l nodes=4:single3200
requests 4 nodes with 3.2 GHz single-core processorson,
#PBS -l nodes=3:ppn=4:dual
requests 4 processors in each of 3 nodes with dual-core processors, and
#PBS -l nodes=2:ppn=8:quad
requests 8 processors in each of 2 nodes with quad-core processors. Note that the "ppn" specification really refers to processing cores, so that "ppn=4" is satisfied by two dual-core processors in a node.
[edit] Help! My Job Won't Run!
[edit] My job won't run. In fact, I get an error message from a "cd" command in my PBS script that is just changing to a directory that I know exists. What is wrong?
It is very likely that you either uploaded the PBS script from a Windows computer or used a cut-and-paste operation to copy it from a Windows computer to the master node of the cluster. Windows computers use a different character sequence (\r\n) to mark the ends of lines and this sequence is not properly handled on the cluster, which uses a single character (\n) to mark the end of a line. You can type in the PBS script from scratch on the master node, taking care not to use any of the original script file. Or you can use the following command on the cluster to strip out the offending carriage return (\r) characters:
dos2unix my_script.pbs
[edit] My job won't run. I get the following error message:
[0][mpid/ch_vapi/viainit.c:129] error(-243): cannot get HCA handle [1][mpid/ch_vapi/viainit.c:129] error(-243): cannot get HCA handle mpiexec: Warning: tasks 0-1 exited with status 254.
[edit] What is going on?
Those error messages are indicative of a job trying to use the Infiniband interconnect on a node without an Infiniband adapter. Since we do not currently have Infiniband capability on any nodes, your program will have to be recompiled for the Gigabit Ethernet interconnect.
[edit] The job I submitted is listed by the "showq" command in the "blocked" list with a state of "BatchHold". What does this mean?
Your job would be placed in that state because the resource manager was repeatedly unable to satisfy resource requests. You should check your PBS script to ensure that it does not request resources that are impossible to obtain. If you're unable to determine the reason for the "BatchHold" state, please contact its-cluster-admin@case.edu for assistance.
[edit] My job enters the queue successfully, but it waits a long time before it gets to run. How can I make my jobs spend less time waiting for cluster time?
The job scheduling software on the cluster makes decisions about how best to allocate the cluster nodes to individual jobs and users. There are ways to make your job more likely to get nodes faster.
- Don't ask for more time than you need. Using commands within your job submission file, it is possible to trace the real start time and end time of your job, and compare them to the original amount of time you asked for from the job scheduler. If your job asks for much more time than needed, the job scheduler will think it cannot provide computers when in reality the job could have fit into a timeslot successfully. This ties up cluster resources needlessly, and means your job could have been done already. Calling the date command from within job submission script, once just before calling the actual computation part of the job submission script, and again at the end of the script will place two timestamps into the job output file. Use this data as a sanity check against how much time to ask for in future job submissions of similar calculations. Some job scripts have asked for more than double the time the job actually requires, and receive longer queue wait times as a result.
- Don't ask for more processors than you need. This may be something that cannot be changed, depending on the nature of the job or simulation. Some simulations require several processors. However, the scheduler has to take certain computers out of the pool and hold them unused until a full set of machines can be assigned to a job asking for several computers. Such a job may have to wait for many days before an opportunity is granted by the job scheduler.
- Debug small, get it right, then scale it up.When first setting up a job, if possible, ask for just a single processor for a short amount of time to ensure your script has no syntax errors or other problems. If the simulation is working properly and the clock expires on the job, the job scheduler will take care of stopping the job. Then try again with a short amount of time and two processors. If it still works, scale up the simulation to a larger data size or more processors. Doing this every time a new kind of job is requested will allow the user to get quick feedback from the cluster, and not waste days waiting to find out about a silly misplaced program argument. It is a good idea to also use this debugging period to properly size the amount of time for the scaled-up job request.
[edit] X Forwarding
The ITS HPC cluster uses XForwarding over ssh to provide support for software with a graphical user interface (GUI). To use software with a GUI, the machine attaching to the cluster must provide an XServer. Case affiliates using Windows are encouraged to download WinaXe, a commercial XServer for Windows. WinaXe is available from the CWRU software center. WinaXe instructions
Note that to attach to the cluster, the user will have to configure an ssh client to allow XForwarding. This step is dependent on the client in use. Please consult documentation for your ssh client on how to perform this step.
Most cluster applications will perform properly with the default settings found in WinaXe. However, Matlab requires changes to the WinaXe settings. Launch the WinaXe application called XSettings, which will be in the same WinaXe group in your Start menu Application list. Adjust your XSetting values to match the image below.
After making the changes, press OK to save your changes, and if prompted to restart WinaXe, say yes. Note that as of March 23, 2007, there is a known issue with XForwarding for Matlab R2006a, and the ITS cluster admins recommend instead using Matlab R2006b. To change versions of matlab issue the commands:
module unload matlab
module load matlab_R2006b_serial
now launch Matlab with the command
matlab
as normal.
Running Mathematica as an X client on the cluster currently requires installing Mathematica fonts on the local machine. Users running Windows may have more success installing one of the free X Servers Xmingand Cygwin/X. See Mathematica Font Setup for installation instructions.
[edit] Miscellaneous
[edit] My job does a lot of I/O. What can I do to improve I/O performance?
Without modifying the software that actually carries out the I/O operations, the best way to optimize I/O operations is to ensure that the disk being accessed in the I/O operations is in the same compute node as the processor executing the I/O operations.
Each compute node contains a hard drive with a /tmp directory that can be used to hold files while jobs accessing them are executing. Placing files used by your job in this /tmp directory will permit the files to be accessed by the job without using the cluster interconnect. When the job completes any files created or modified by the job must be copied from /tmp to your home directory.
To use the local drive within a compute note, you should create (or, more accurately, have your batch job create) a separate directory within /tmp on the local drive to hold the files for the job and then copy the necessary files to this directory before running the job. When the job completes, the appropriate output files can be copied from this temporary directory back to the home directory.
An example follows:
mkdir /tmp/$USER.$JOBID # create a temporary directory for this job cp myfile1 myfile2 /tmp/$USER.$JOBID # copy files to the temporary directory cd /tmp/$USER.$JOBID # change to the temporary directory module unload module_I_dont_want # unload conflicting modules module load module_I_want # load needed modules mpiexec myprogram < inputfile > outputfile # run the program cp outputfile otherfile $HOME # copy output files to home directory cd $HOME # change back to home directory rm –rf /tmp/$USER.$JOBID # delete the temporary directory exit # explicitly terminate the job – not absolutely necessary
Typically these commands would be placed in a PBS batch script and run. In this case, a simpler alternative exists as illustrated in this example:
cp myfile1 myfile2 $TMPDIR # copy files to the temporary directory cd $TMPDIR # change to the temporary directory module unload module_I_dont_want # unload conflicting modules module load module_I_want # load needed modules mpiexec myprogram < inputfile > outputfile # run the program cp outputfile otherfile $HOME # copy output files to home directory exit # explicitly terminate the job – not absolutely necessary
Note that the temporary directory $TMPDIR is automatically created and deleted when the job completes.
[edit] What applications software is available on the cluster?
Please see the section on “Applications” at ITS_High_Performance_Computing_Cluster for current information.
[edit] Are there any tutorials or other documentation on using the cluster?
See http://www.osc.edu/hpc/computing/p4 for the programming environment. See http://www.osc.edu/hpc/software/intelP4/index.shtml for available software with links for more information. Much of the information available at http://www.osc.edu pertaining to their Pentium 4 cluster is relevant to the Case cluster.
[edit] When I try to ssh to a node I get the following message. What's wrong and how do I fix it?
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is b9:e0:3d:63:c3:d7:93:7b:cd:c1:9d:32:3a:a6:f8:58. Please contact your system administrator. Add correct host key in /home/abc123/.ssh/known_hosts to get rid of this message. Offending key in /home/abc123/.ssh/known_hosts:17 RSA host key for node19 has changed and you have requested strict checking. Host key verification failed.
This will happen from time to time when a compute node is rebooted or another change is made to the cluster configuration. The solution is very simple: use a standard editor (e.g., vi) on the master node to edit the file /home/abc123/.ssh/known_hosts and delete the line that corresponds to the node mentioned in the error message - above it is "node19". The line will be very long and will begin with the node name, like
node19,192.168.252.119 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAy2EtXw+uYkFLPXiDxsQ2tMjXMyYzLZiWd3foLL1/UhIasMoNDLASD2wfI4pOFWPp5 CKZwzhQWGbquLqHxyJpUvRORoXQ2AT9zZxYQHBvqNpPJpl/JlMRrh5Y741Aw/vQ4KRHRiB+PiaCVlVvV+vYYZECFkLgZiQ1kfvLNuSNMTM=
Just delete the line, save the file, and try again.
[edit] Your question isn't answered here?
Please send it to its-cluster-admin@case.edu for a personal reply and inclusion in the FAQ in the future.
| ITS High Performance Cluster Articles | |
|---|---|
| This article is part of the ITS Cluster series of articles | |
| FAQ | Intel compilers | GNU compilers | Portland Group compilers | |
| Intel Math Kernel Library | MINPACK | ScaLAPACK | GSL | FFTW3 | MPICH | NAG | |
| R | Mathematica | NAMD | GROMACS | Amber | MATLAB | |
| FLUENT | GAMESS | Gaussian | MOLCAS | LAMMPS | APBS | |
Category: ITS High Performance Cluster
Departments > ITS
Services > IT Services > ITS Services > ITS High Performance Cluster
Case Referrers
Other Sites
- The ITS High Performance Computing Cluster (510 referral)
- Pilot HPC Cluster (8 referral)
- http://start.case.edu/ (27 referral)
- Home of Research Computing (1 referral)
- Departments - ITS (1 referral)