HiperGator user agreement

Acceptable Use Policy

• ACCEPTABLE USE

I acknowledge that the access to the HPC resources operated by UF Research Computing is subject to the UF Acceptable Use Policy at https://it.ufl.edu/policies/acceptable-use/acceptable-use-policy/ and the Research Computing policies at http://www.rc.ufl.edu/services/procedures/ and that I am responsible for following these policies.

• RESTRICTED DATA

I also certify that using restricted data and software on the HPC resources requires extra steps described at UFRC Policies and at UFRC Export Policies, and that I will notify both my account sponsor and the Office of Research (Research Compliance) and Research Computing at support.rc.ufl.edu when I am working with such data.

Basic usage of HiperGator

• Interactive sessions on HiperGator
• Job submissions on HiperGator

These usages can be applied to other Linux machines.

Preparation

• Softwares
  • FileZilla
  • terminal (Mac, Linux)
  • MobaXterm (Windows)
• scripts:
  • HiperGator: /blue/phc6068/share/zhuo/example

How to login HiperGator (Windows)

• MobaXterm
  • ssh user@hpg.rc.ufl.edu
  • enter your password

How to login HiperGator (Mac, Linux)

• terminal
  • ssh user@hpg.rc.ufl.edu
  • enter your password

Your HiperGator Home directory

• /home/yourUserName
  • same as ~
  • 40GB space
• /blue/phc6068/yourUserName
  • This one will be temporary, will be deleted at the end of this sememster
  • If you already have hiperGator sponser
    • You can create a folder under /blue/phc6068/share
    • You can replace phc6068 with your sponser’s group name
  • The entire phc6068 group share 2TB space

Common linux commands:

• pwd: currect working directory
• ls: list all files in current working directory
• mkdir afolder: create a folder named afolder
• cd afolder: get into afolder
• cd ..: go back one folder
• cd ~: go to your home directory
• top: monitor CPU/memory usages

Common linux commands:

• cd /blue/phc6068/share/zhuo/example/testR ## go the the folder with all test files
• cp saveCars.R /blue/phc6068/your directory ## copy a single file
• cp -r /blue/phc6068/share/zhuo/example /blue/phc6068/your-directory/. ## copy the entire folder
• cat saveCars.R: print everying
• more saveCars.R: interactive print
• head saveCars.R: head of the contents
• cat saveCars.slurm:
• rm saveCars.slurm: delete this file

FileZilla

FileZilla

• Host: hpg.rc.ufl.edu
• username:
• password:
• Port: 22
• connect

You can transfer files between your local computer and hiperGator

Login node and working node

Login node and working node

• You are not allowed to do any computing at the login node
  • If the login node crushes, nobody can use hiperGator
  • The HiperGator team also monitors this and may kill your jobs on the login node
• Two ways to access the working nodes
  • Interactive session
  • Submit slurm jobs.

Open an interactive session on HiperGator (1)

## open interactive R session
module load ufrc
srundev  --account=phc6068 --qos=phc6068 --time=04:00:00

module load R ## load R
R

Open an interactive session on HiperGator (2)

## open interactive R session
srun --account=phc6068 --qos=phc6068 --ntasks=1 --cpus-per-task 1 --mem=8gb  --time=04:00:00 --pty bash -i

module load R ## load R

R

• this is shell script
• need to specify account and qos
• ntasks: number of machines (nodes)
• cpus-per-task: number of CPUs
• mem: memory you want to claim. The entire class shares 112G.
• time: specifies how long you want to claim for the interactive session.
• -i specifies interactive session.
• module load R: will load R software
• R: open R software

Open an interactive R session on HiperGator

## do the following on hiperGator
getwd()
dir()

head(cars)
mycars <- cars
write.csv(mycars, "mycars.csv")
dir()

• Try to transfer the file back to local PC using FileZilla

Submit a job (I)

1. R script (saveCars.R): contains your R code
2. SLURM job script (saveCars.slurm): coordinate your job with the server
3. submit: sbatch the slurm file

Prepare R script (a simple one) (I)

• Save the following code into saveCars.R

WD <- "/blue/phc6068/share/zhuo/example/testR" ## change to your own directory
dir.create(WD, re=T) ## force to create this folder

setwd(WD) ## set to your own directory!
mycars <- mtcars
write.csv(mycars,"mycars.csv")

Prepare SLURM job script (I)

• Save the following code into saveCars.slurm

#!/bin/sh
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --account=phc6068             # your own sponser or account from this class
#SBATCH --qos=phc6068                 # your own sponser or account from this class
#SBATCH --mail-type=ALL               # Mail events
#SBATCH --mail-user=xx@xx.xx          # Where to send email 
#SBATCH --ntasks=1                    # Run on a single machine (node)
#SBATCH --cpus-per-task 1             # Run on a single CPU
#SBATCH --mem=8gb                     # Memory limit
#SBATCH --time=04:00:00               # Time: hrs:min:sec
#SBATCH --output=serial_test_%j.out   # Output and error log 

pwd; hostname; date 

module load R 

echo "Running save cars script on a single CPU core" 

R CMD BATCH saveCars.R ## make sure saveCars.R is at your current working directory
## R --no-save --quiet --slave < saveCars.R ## alternative way

date

submit the job (I)

cd /blue/phc6068/share/zhuo/example/testR
sbatch saveCars.slurm ## submit job

1. Submit the slurm job (saveCars.slurm)
2. The slurm job contains the code to run the R job (R CMD BATCH saveCars.R)
3. The R job will return the result

Check log file (I)

• Linux log file:

cd /blue/phc6068/share/zhuo/example/testR
cat serial_test_25280301.out ## you may have your own log file name
head serial_test_25280301.out ## you may have your own log file name
more serial_test_25280301.out ## you may have your own log file name

• R console history

cd /blue/phc6068/share/zhuo/example/testR
cat saveCars.Rout

• R/Slurm job output

cd /blue/phc6068/share/zhuo/example/testR
cat mycars.csv

Exercise (I)

1. Copy the saveCars.R and saveCars.slurm into your own working directory
2. Revise the saveCars.R
   • Transfer to your local computer using Filezilla
   • Change to your own working directory
   • Transfer back to hiperGator
3. Try to revise the saveCars.slurm
   • Try to specify your email, revise time and memory
4. Submit your job

Submit a job with external argument (II)

• R script (saveCarsArgs.R): contains your R code
• SLURM job script (saveCarsArgs.slurm): coordinate your job with the server
• submit: sbatch the slurm file

R script (with external arguments) (II)

• Save the following code into saveCarsArgs.R

args = commandArgs(trailingOnly = TRUE) ## pass in external argument

rowID <- args[1]
aarg <- as.numeric(rowID)

setwd("/blue/phc6068/share/zhuo/example/testR2")
mycars <- mtcars[aarg,]
filename <- paste0("arg",aarg,".csv")
write.csv(mycars,filename)

SLURM job script (II)

• Save the following code into saveCarsArgs.slurm

#!/bin/sh
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --account=phc6068             # your own sponser or account from this class
#SBATCH --qos=phc6068                 # your own sponser or account from this class
#SBATCH --mail-type=ALL               # Mail events
#SBATCH --mail-user=xx@xx.xx          # Where to send email 
#SBATCH --ntasks=1                    # Run on a single machine (node)
#SBATCH --cpus-per-task 1             # Run on a single CPU
#SBATCH --mem=8gb                    # Memory limit
#SBATCH --time=04:00:00               # Time: hrs:min:sec
#SBATCH --output=serial_test_%j.out   # Output and error log 

pwd; hostname; date 

module load R 

echo "Running save cars script on a single CPU core" 

R --no-save --quiet --slave --args 1 < saveCarsArgs.R ## again, make sure the R file is at your current working directory.

date

submit the job (II)

cd /blue/phc6068/share/zhuo/example/testR2
sbatch saveCarsArgs.slurm ## submit job

1. Submit the slurm job (saveCarsArgs.slurm)
2. The slurm job will submit the R job (saveCarsArgs.R) with extra argument
3. The R job will return the result

Check log file (II)

• Linux log file:

cd /blue/phc6068/share/zhuo/example/testR2
cat serial_test_25280860.out ## you may have your own log file name

• R console history
  • –no-save: no R console history

  R --no-save --quiet --slave --args 1 < saveCarsArgs.R 

• R/Slurm job output

cd /blue/phc6068/share/zhuo/example/testR2
cat arg1.csv

Submit a job with loops (III)

• R script (saveCarsArgs.R): save as (II)
• SLURM job script (saveCarsArgsLoops.slurm): coordinate your job with the server
• submit: sbatch the slurm file

SLURM job script (III)

• Save the following code into saveCarsArgsLoops.slurm

#!/bin/sh
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --account=phc6068             # your own sponser or account from this class
#SBATCH --qos=phc6068                 # your own sponser or account from this class
#SBATCH --mail-type=ALL               # Mail events
#SBATCH --mail-user=xx@xx.xx          # Where to send email 
#SBATCH --ntasks=1                    # Run on a single machine (node)
#SBATCH --cpus-per-task 1             # Run on a single CPU
#SBATCH --mem=8gb                    # Memory limit
#SBATCH --time=04:00:00               # Time: hrs:min:sec
#SBATCH --output=serial_test_%j.out   # Output and error log 

pwd; hostname; date 

module load R 

for i in {2..10}
do
echo "Running save cars" $i 
R --no-save --quiet --slave --args $i < saveCarsArgs.R 
done

date

submit the job (III)

cd /blue/phc6068/share/zhuo/example/testR3
sbatch saveCarsArgsLoops.slurm ## submit a loop job

Check job status

• squeue: all jobs from all users on hiperGator
• squeue | grep zhuo: all jobs from user zhuo on hiperGator ## replace zhuo with your GatorLinkID
• squeue -A phc6068: all jobs under group phc6068
• scancel jobID: cancel a job with the jobID

Burst mode

• Guaranteed computing resources
  • qos=phc6068
  • 112GB
  • 32 CPUs

Since our entire class share these resouces, please plan ahead when you work on your HWs. Using the hiperGator at the last minute may make you compete with other students for resources.

If all computing resources are occupied

• Burst mode (9 times of original number of CPUs & memory)
  • qos=phc6068-b
  • Use idle resources, depending on other groups

recommended text editor

• Mac: TextMate
• Windows: https://www.techtimes.com/brandspin/240045/20190325/top-5-best-text-editor-windows-2019.htm