SCC Batch Computing
Introduction
Most of your heavy compute, such as model training is best run as non-interactive batch jobs on the SCC.
Here, we’ll get familiar with the process and commands.
SCC’s batch job system is based on the Sun Grid Engine
References:
https://www.bu.edu/tech/support/research/system-usage/running-jobs/
All the examples here need to be run on an SCC node. A login node is fine.
qsub command
Non-interactive batch jobs are submitted with the qsub command.
Here’s a trivial batch job submission. We’ll just execute the shell printenv command which prints all the environment variables in your shell.
$ qsub -b y printenv
Your job 294013 ("printenv") has been submittedThe -b y option says the command is likely a binary executable, but can also be a script. The submission host won’t try to parse the command as a script and will just pass the path to the command to the execustion node.
qstat command
You can check the status of your job in the queue with
$ qstat -u tgardos
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
2949274 0.00000 printenv tgardos qw 02/04/2026 14:47:55 The -u userID option only shows your jobs. Leave it off and you will see all the jobs currently queued.
You see it lists the
- job ID
- priority of the job
- name of the job
- user
- job’s state in the queue – here it is
qw(waiting to run). - submission/start time
- queue – the queue name if the job is running
At some point qstat will no longer show any jobs in the queue. This one doesn’t run for long, so you probably won’t catch it running, but if you do, it will look like this:
$ qstat -u tgardos
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
2949274 0.74554 printenv tgardos r 02/04/2026 14:49:56 academic@scc-gf1.scc.bu.edu 1 After the job finishes, you’ll see in the directory you submitted the job from 2 files:
$ ls -l
total 1
-rw-r--r-- 1 tgardos dl4ds-ta 0 Feb 4 14:49 printenv.e2949274
-rw-r--r-- 1 tgardos dl4ds-ta 2062 Feb 4 14:49 printenv.o2949274Notice the files are the job name with either a .e<jobID> or .o<jobID> output.
These are standard and error outputs of the job.
In this case the error output was 0 bytes, but the job output was
$ cat printenv.o294013
QUEUE=cds
SGE_O_HOST=scc1
HOSTNAME=scc-tc4.scc.bu.edu
ENVIRONMENT=BATCH
REQUEST=printenv
SGE_STDIN_PATH=/dev/null
SGE_CELL=default
NHOSTS=1
SGE_O_WORKDIR=/usr2/faculty/tgardos
...Which is the printout of the environment variables on the node the job was executed on, which was scc-tc4.scc.bu.edu.
You’ll also see a lot environment variabels with SGE_ prefix, which are apparently set as part of the Sun Grid Engine batch queueing system.
qacct command
You can get information about completed jobs with
$ qacct -j 2949274
==============================================================
qname engineering
hostname scc-pi6.scc.bu.edu
group mhcpep
owner gychuang
project mhcpep
department defaultdepartment
jobname sge.jcf
jobnumber 2949274
taskid undefined
account sge
priority 0
qsub_time Tue Mar 4 00:25:36 2025
start_time Tue Mar 4 00:27:53 2025
end_time Tue Mar 4 00:27:54 2025
granted_pe mpi4
slots 4
failed 0
exit_status 1
ru_wallclock 1
ru_utime 0.207
ru_stime 0.111
ru_maxrss 19576
ru_ixrss 0
ru_ismrss 0
ru_idrss 0
ru_isrss 0
ru_minflt 18585
ru_majflt 0
ru_nswap 0
ru_inblock 0
ru_oublock 40
ru_msgsnd 0
ru_msgrcv 0
ru_nsignals 0
ru_nvcsw 1786
ru_nivcsw 38
cpu 0.318
mem 0.000
io 0.000
iow 0.000
maxvmem 0.000
arid undefined
==============================================================
qname academic
hostname scc-gf1.scc.bu.edu
group dl4ds
owner tgardos
project dl4ds
department defaultdepartment
jobname printenv
jobnumber 2949274
taskid undefined
account sge
priority 0
qsub_time Wed Feb 4 14:47:55 2026
start_time Wed Feb 4 14:49:56 2026
end_time Wed Feb 4 14:49:56 2026
granted_pe NONE
slots 1
failed 0
exit_status 0
ru_wallclock 0
ru_utime 0.008
ru_stime 0.014
ru_maxrss 3676
ru_ixrss 0
ru_ismrss 0
ru_idrss 0
ru_isrss 0
ru_minflt 777
ru_majflt 0
ru_nswap 0
ru_inblock 8
ru_oublock 24
ru_msgsnd 0
ru_msgrcv 0
ru_nsignals 0
ru_nvcsw 64
ru_nivcsw 0
cpu 0.022
mem 0.000
io 0.000
iow 0.000
maxvmem 0.000
arid undefinedWe’ll come back to why ‘department’ and ‘project’ maybe impact which resources have access to.
qdel command
You can delete a job waiting in queue with qdel jobID command.
Other Interactive Tools
We’re going to be playing with some short scripts. You can do everything via the SCC Dashboard login node and text editors, but there are alternatives:
- ssh into a login node,
ssh <username>@scc1.bu.edu - start an interactive VS Code server
- start an interactive desktop and open VS Code or terminal
It’s also to use a local instance of VS Code with Remote Extension Pack.
Note: This is not officially supported by SCC and has the annoying problem of having to re-connect to the remote host every time you restart VS Code.
- Open VS Code locally on your machine.
- Install the “Remote Development” extension pack by Microsoft from the extension marketplace.
- Now connect to remote host
scc1.bu.edu, follow the prompts to connect. - From the Terminal menu, open a new terminal.
Submitting a script
In general we don’t want to run a single binary command but rather usually a python script with some environment configuration first.
For that it’s best to submit a shell script that configures the environment and then runs the python script.
So let’s create a shell script
scriptv1.sh
#!/bin/bash -l
echo "Print python version"
python --version
python myscript.py
Important
To be processed correctly, the shell script must have a blank line at the end of the file.
You see that it is running a python script, that in this case can be a simple script
myscript.py
print("Hello batched world!")We can submit and check the status.
$ qsub scriptv1.sh
Your job 294119 ("scriptv1.sh") has been submitted
$ qstat -u tgardos
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
294119 0.00000 scriptv1.sh tgardos qw 09/21/2024 21:12:46 1 Actually, in this case, I caught the queue status while it was running
$ qstat -u tgardos
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
294119 1.10000 scriptv1.sh tgardos r 09/21/2024 21:14:04 cds@scc-tc3.scc.bu.edu 1 And cat the output
$ cat scriptv1.sh.o294119
Print python version
Python 3.6.8
Hello batched world!
Tip
By the way, even for longer jobs, I will start the script manually just to make sure it starts ok, then kill it, instead of waiting for it to queueu and run to find out I made a simple mistake in sthe script.
Caution
This won’t work if you want to train on a GPU, but you can use the qrsh command to get a GPU node.
$ source scriptv1.sh
Print python version
Python 3.6.8
Hello batched world!Setting up batch job environment
So one thing you’ll notice is that the python version is 3.6.8, which is not the latest version. So let’ update our script.
scriptv2.sh
#!/bin/bash -l
module load python3/3.12.4
echo "Print python version"
python --version
python myscript.pyNow when we run
$ source scriptv2.sh
Print python version
Python 3.12.4
Hello batched world!Job Submission Directives
We can add job submission directives to our shell script with lines beginning with #$.
scriptv3.sh
#!/bin/bash -l
#$ -P dl4ds # Assign to project dl4ds
#$ -l h_rt=12:00:00 # Set a hard runtime limit
#$ -N hello-world # Give the job a name other than the shell script name
#$ -j y # merge the error and regular output into a single file
module load python3/3.12.4
echo "Print python version"
python --version
python myscript.pyOther general job submission directives are listed here
Requesting Resources
Let’s look at how to request particular resources for our job.
Here’s simply python code that checks if a GPU is available, and if so transforms a simple tensor into a CUDA tensor.
Note
CUDA is the Nvidia GPU driver and software package.
cuda-simple.py
import torch
print(f'torch cuda is available: {torch.cuda.is_available()}')
t = torch.tensor([1, 2, 3])
if torch.cuda.is_available():
t = t.cuda()
print(t)And we create a job submission script that loads the SCC academic ML environment and activates it.
run-cuda-simple.sh
#!/bin/bash -l
#$ -P dl4ds # Assign to project dl4ds
#$ -j y # merge the error and regular output into a single file
module load miniconda academic-ml/spring-2026
conda activate spring-2026-pyt
echo "Print python version"
python --version
python cuda-simple.py
# to be processed correctly there must be a blank line at the end of the fileLet’s submit two jobs:
One to run on default compute node (no GPU)
$ qsub run-cuda-simple.sh
Your job 316551 ("run-cuda-simple.sh") has been submittedAnd one job where we request 1 GPU.
$ qsub -l gpus=1 run-cuda-simple.sh
Your job 316561 ("run-cuda-simple.sh") has been submittedIn this case, it created an output file for each job:
ls -ls
run-cuda-simple.sh.o316551
run-cuda-simple.sh.o316561Let’s look at the one where we didn’t request a GPU.
run-cuda-simple.sh.o316551
-------------------------------------------------------------------------------
To activate the conda environment in a batch script or in a terminal run the
following command. For interactive sessions in SCC OnDemand place this command
in the "Pre-Launch Command" field.
To load the PyTorch-based environment run:
conda activate fall-2024-pyt
To load the Tensorflow-based environment run:
conda activate fall-2024-tf
To load the Jax-based environment run:
conda activate fall-2024-jax
For information on using or cloning this conda environment visit:
https://www.bu.edu/tech/support/research/software-and-programming/common-languages/python/python-ml/academic-machine-learning-environment
-------------------------------------------------------------------------------
Print python version
Python 3.11.9
torch cuda is available: False
tensor([1, 2, 3])And the one where we did.
run-cuda-simple.sh.0316561
-------------------------------------------------------------------------------
To activate the conda environment in a batch script or in a terminal run the
following command. For interactive sessions in SCC OnDemand place this command
in the "Pre-Launch Command" field.
To load the PyTorch-based environment run:
conda activate fall-2024-pyt
To load the Tensorflow-based environment run:
conda activate fall-2024-tf
To load the Jax-based environment run:
conda activate fall-2024-jax
For information on using or cloning this conda environment visit:
https://www.bu.edu/tech/support/research/software-and-programming/common-languages/python/python-ml/academic-machine-learning-environment
-------------------------------------------------------------------------------
Print python version
Python 3.11.9
torch cuda is available: True
tensor([1, 2, 3], device='cuda:0')You can see here for a more list of resources to specify, but a good example is
$ qsub -l gpus=1 -l gpu_c=7.0 -pe omp 8 script.shThe argument -l qpu_c=7.0 is the GPU capability requested, which currently is one of [3.5, 5, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6, 8.9, 9.0].
There’s a dedicated page for GPU Computing on the SCC.
You can list all installed GPUs with qgpus.
# Run Feb 4, 2026
$ qgpus
gpu_type total in_use available
-------- ----- ------ ---------
A100 5 0 5
A100-80G 24 21 3
A40 68 31 35
A6000 76 24 52
H200 20 20 0
K40m 14 2 12
L40 6 2 4
L40S 118 87 30
P100 28 10 18
P100-16G 23 4 19
RTX6000 5 0 5
RTX6000ada 30 18 12
RTX8000 8 2 6
TitanV 8 0 8
TitanXp 10 0 10
V100 65 33 31
V100-32G 4 3 1To see more details, the resource number and which queues they’re on, you can use qgpus -v.
# Run on Feb 4, 2026
$ qgpus -v
host gpu_type gpu_c gpu_mem cpu_ cpu_ gpu_ gpu_ gpu_ queue_list
total in_use total in_use avail
-------- -------- ----- ------- ----- ------ ----- ------ ----- ------------------------------
scc-205 A100 8.0 40G 32 0 4 0 4 thinfilament-gpu,thinfilament-gpu-pub
scc-207 A100 8.0 40G 32 1 1 0 1 li-rbsp-gpu-pub,li-rbsp-gpu
scc-210 A100 8.0 80G 32 32 4 4 0 neuro-autonomy,neuro-autonomy-pub
scc-212 A100 8.0 80G 32 7 4 4 0 a100
scc-219 A100 8.0 80G 32 12 2 1 1 joshigroup-gpu,joshigroup-gpu-pub
scc-220 A100 8.0 80G 32 16 4 4 0 labcigroup-gpu-pub,labcigroup-gpu
scc-221 A100 8.0 80G 32 0 2 0 2 chapmangroup-gpu,chapmangroup-gpu-pub
scc-305 A100 8.0 80G 48 48 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-306 A100 8.0 80G 48 48 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-211 A40 8.6 48G 32 10 4 4 0 a40
scc-213 A40 8.6 48G 32 0 2 0 2 csgpu-pub,csgpu
scc-214 A40 8.6 48G 32 16 4 4 0 ece-pub,ece,ece-long
scc-215 A40 8.6 48G 32 0 4 0 4 ece-pub,ece,ece-long
scc-216 A40 8.6 48G 32 0 4 0 4 ece-pub,ece,ece-long
scc-217 A40 8.6 48G 32 4 2 1 1 csgpu-pub,csgpu
scc-218 A40 8.6 48G 32 16 2 2 0 csgpu-pub,csgpu
scc-301 A40 8.6 48G 32 0 6 0 6 neuro-autonomy,neuro-autonomy-pub
scc-302 A40 8.6 48G 32 32 6 4 0 neuro-autonomy,neuro-autonomy-pub
scc-303 A40 8.6 48G 32 0 10 0 10 iris-gpu,iris-gpu-pub
scc-f03 A40 8.6 48G 32 0 6 0 6 neuro-autonomy,neuro-autonomy-pub
scc-f04 A40 8.6 48G 32 22 10 10 0 ivcbuyin-long,ivcbuyin,ivcbuyin-pub
scc-f05 A40 8.6 48G 32 10 8 6 2 ivcbuyin,ivcbuyin-pub
scc-307 A6000 8.6 48G 32 0 4 0 4 batcomputer,batcomputer-pub
scc-603 A6000 8.6 48G 32 4 5 1 4 iris-gpu,iris-gpu-pub
scc-604 A6000 8.6 48G 32 0 8 0 8 iris-gpu,iris-gpu-pub
scc-605 A6000 8.6 48G 32 0 8 0 8 iris-gpu,iris-gpu-pub
scc-606 A6000 8.6 48G 32 0 8 0 8 iris-gpu,iris-gpu-pub
scc-607 A6000 8.6 48G 32 0 6 0 6 iris-gpu,iris-gpu-pub
scc-608 A6000 8.6 48G 32 0 8 0 8 iris-gpu,iris-gpu-pub
scc-e01 A6000 8.6 48G 32 20 9 5 4 ivcbuyin,ivcbuyin-pub
scc-e02 A6000 8.6 48G 32 24 10 10 0 ivcbuyin,ivcbuyin-pub
scc-e03 A6000 8.6 48G 32 12 10 8 2 ivcbuyin,ivcbuyin-pub
scc-a03 H200 9.0 144G 32 32 4 4 0 h200
scc-a04 H200 9.0 144G 32 17 4 4 0 h200
scc-a05 H200 9.0 144G 32 28 4 4 0 ece-pub,ece,ece-long
scc-a06 H200 9.0 144G 32 32 4 4 0 ece-pub,ece,ece-long
scc-a15 H200 9.0 144G 32 32 2 2 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-a16 H200 9.0 144G 32 32 2 2 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-c01 K40m 3.5 12G 20 3 2 2 0 katia,k40
scc-c04 K40m 3.5 12G 20 0 4 0 4 kulisgpu,kulisgpu-pub
scc-c05 K40m 3.5 12G 20 0 4 0 4 kulisgpu,kulisgpu-pub
scc-sc1 K40m 3.5 12G 16 0 2 0 2 tcn-pub,tcn
scc-sc2 K40m 3.5 12G 16 0 2 0 2 tcn-pub,tcn
scc-304 L40 8.9 48G 32 8 6 2 4 ivcbuyin,ivcbuyin-pub
scc-501 L40S 8.9 48G 32 17 4 4 0 l40s
scc-502 L40S 8.9 48G 32 32 4 3 0 l40s
scc-503 L40S 8.9 48G 32 10 4 4 0 l40s
scc-504 L40S 8.9 48G 32 15 4 4 0 l40s
scc-505 L40S 8.9 48G 32 28 4 4 0 l40s
scc-506 L40S 8.9 48G 32 0 4 0 4 l40s
scc-507 L40S 8.9 48G 32 20 2 2 0 czcb-buyin,czcb-buyin-pub
scc-508 L40S 8.9 48G 32 0 2 0 2 csgpu-pub,csgpu
scc-509 L40S 8.9 48G 32 4 2 1 1 csgpu-pub,csgpu
scc-510 L40S 8.9 48G 32 17 4 4 0 l40s
scc-511 L40S 8.9 48G 32 21 4 4 0 l40s
scc-512 L40S 8.9 48G 32 21 4 4 0 l40s
scc-513 L40S 8.9 48G 32 28 4 4 0 l40s
scc-601 L40S 8.9 48G 32 24 8 6 2 cuigpu,cuigpu-pub
scc-a01 L40S 8.9 48G 32 28 8 7 1 cuigpu,cuigpu-pub
scc-a02 L40S 8.9 48G 32 20 8 5 3 cuigpu,cuigpu-pub
scc-j01 L40S 8.9 48G 32 8 4 1 3 ece-pub,ece,ece-long
scc-j02 L40S 8.9 48G 32 8 4 1 3 ece-pub,ece,ece-long
scc-j03 L40S 8.9 48G 32 12 4 2 2 ece-pub,ece,ece-long
scc-j04 L40S 8.9 48G 32 16 4 1 3 ece-pub,ece,ece-long
scc-j05 L40S 8.9 48G 32 8 4 2 2 ece-pub,ece,ece-long
scc-j06 L40S 8.9 48G 32 24 4 4 0 ece-pub,ece,ece-long
scc-j07 L40S 8.9 48G 32 4 4 4 0 ece-pub,ece,ece-long
scc-j08 L40S 8.9 48G 32 20 4 3 1 ece-pub,ece,ece-long
scc-j09 L40S 8.9 48G 32 0 2 0 2 csgpu-pub,csgpu
scc-j10 L40S 8.9 48G 32 4 2 1 1 csgpu-pub,csgpu
scc-j11 L40S 8.9 48G 32 17 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-j12 L40S 8.9 48G 32 16 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-j13 L40S 8.9 48G 32 16 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-c08 P100 6.0 12G 28 8 2 1 1 p100
scc-c09 P100 6.0 12G 28 17 2 2 0 p100
scc-c10 P100 6.0 12G 28 12 2 2 0 p100
scc-c11 P100 6.0 12G 28 24 2 2 0 p100
scc-c12 P100 6.0 12G 28 4 4 1 3 csgpu-pub,csgpu
scc-c13 P100 6.0 12G 28 0 4 0 4 ece-pub,ece,ece-long
scc-c14 P100 6.0 12G 28 0 4 0 4 ece-pub,ece,ece-long
scc-x01 P100 6.0 12G 28 16 2 2 0 onrcc-gpu,onrcc-gpu-pub
scc-x02 P100 6.0 12G 28 0 2 0 2 onrcc-gpu,onrcc-gpu-pub
scc-x03 P100 6.0 12G 28 0 2 0 2 onrcc-gpu,onrcc-gpu-pub
scc-x04 P100 6.0 12G 28 0 2 0 2 onrcc-gpu,onrcc-gpu-pub
scc-k01 P100 6.0 16G 28 24 4 3 1 ece-pub,ece,ece-long
scc-k02 P100 6.0 16G 28 8 4 1 3 ece-pub,ece,ece-long
scc-k03 P100 6.0 16G 28 0 4 0 4 cuigpu,cuigpu-pub
scc-k04 P100 6.0 16G 28 0 4 0 4 cuigpu,cuigpu-pub
scc-k05 P100 6.0 16G 28 0 4 0 4 cuigpu,cuigpu-pub
scc-k06 P100 6.0 16G 28 0 1 0 1 csdata,csdata-pub
scc-k07 P100 6.0 16G 28 12 2 0 2 bil-koo-gpu,bil-koo-gpu-pub
scc-f02 RTX6000 7.5 24G 32 0 5 0 5 ivcbuyin,ivcbuyin-pub
scc-308 RTX6000ada 8.9 48G 32 24 4 4 0 batcomputer,batcomputer-pub
scc-309 RTX6000ada 8.9 48G 64 25 10 7 3 ivcbuyin,ivcbuyin-pub
scc-602 RTX6000ada 8.9 48G 32 0 8 0 8 cuigpu,cuigpu-pub
scc-609 RTX6000ada 8.9 48G 64 20 8 7 1 ivcbuyin,ivcbuyin-pub
scc-e04 RTX8000 7.5 48G 32 8 8 2 6 ivcbuyin,ivcbuyin-pub
scc-f01 TitanV 7.0 12G 24 0 8 0 8 ivcbuyin-int
scc-e05 TitanXp 6.1 12G 28 0 10 0 10 ivcbuyin,ivcbuyin-pub
scc-201 V100 7.0 16G 32 32 4 4 0 academic-gpu-pub,academic-gpu
scc-202 V100 7.0 16G 32 4 4 1 3 academic-gpu-pub,academic-gpu
scc-203 V100 7.0 16G 32 0 4 0 4 academic-gpu-pub,academic-gpu
scc-204 V100 7.0 16G 32 8 4 2 2 academic-gpu-pub,academic-gpu
scc-k08 V100 7.0 16G 28 4 2 1 1 thinfilament-gpu,thinfilament-gpu-pub
scc-k09 V100 7.0 16G 28 5 2 2 0 casaq-gpu,casaq-gpu-pub
scc-k10 V100 7.0 16G 28 28 1 0 0 bil-koo-gpu,bil-koo-gpu-pub
scc-k11 V100 7.0 16G 28 4 2 1 1 csgpu-pub,csgpu
scc-q23 V100 7.0 16G 32 8 1 1 0 korolevgroup-gpu-pub,korolevgroup-gpu
scc-q24 V100 7.0 16G 32 8 1 1 0 labcigroup-gpu-pub,labcigroup-gpu
scc-q25 V100 7.0 16G 32 0 2 0 2 csgpu-pub,csgpu
scc-q26 V100 7.0 16G 32 0 2 0 2 csgpu-pub,csgpu
scc-q27 V100 7.0 16G 32 16 2 1 1 csgpu-pub,csgpu
scc-q28 V100 7.0 16G 32 0 2 0 2 csgpu-pub,csgpu
scc-q30 V100 7.0 16G 28 16 4 1 3 jchengroup,jchengroup-pub
scc-q31 V100 7.0 16G 32 4 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-q32 V100 7.0 16G 32 4 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-q33 V100 7.0 16G 32 0 4 0 4 ece-pub,ece,ece-long
scc-q34 V100 7.0 16G 32 0 4 0 4 ece-pub,ece,ece-long
scc-q35 V100 7.0 16G 32 20 4 4 0 biophys-gpu-pub,biophys-gpu
scc-q36 V100 7.0 16G 32 12 4 2 2 aclabgroup,aclabgroup-pub
scc-x05 V100 7.0 16G 28 20 2 2 0 v100
scc-x06 V100 7.0 16G 28 2 2 2 0 v100
scc-208 V100 7.0 32G 32 8 2 2 0 thinfilament-gpu,thinfilament-gpu-pub
scc-209 V100 7.0 32G 32 0 1 0 1 cuigpu,cuigpu-pub
scc-q29 V100 7.0 32G 32 4 1 1 0 korolevgroup-gpu-pub,korolevgroup-gpuAnd the GPUs on CDS queues:
# Run on Feb 26, 2026
$ qgpus -v | grep cds
scc-305 A100 8.0 80G 48 48 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-306 A100 8.0 80G 48 48 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-a15 H200 9.0 144G 32 32 2 2 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-a16 H200 9.0 144G 32 32 2 2 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-j11 L40S 8.9 48G 32 17 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-j12 L40S 8.9 48G 32 16 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-j13 L40S 8.9 48G 32 16 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-q31 V100 7.0 16G 32 4 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-long
scc-q32 V100 7.0 16G 32 4 4 4 0 cds-gpu-pub,cds-gpu,cds-gpu-longerExploring Queues
You can explore the queues a bit more with the qselect command.
We can count the total number of queues:
$ qselect | wc -l
1776The number of queues with at least one GPU.
$ qselect -l gpus=1 | wc -l
214We can search for queues with ‘cds’ in the name:
$ qselect -q *cds*
cds-m1024-pub@scc-v08.scc.bu.edu
cds-pub@scc-tb3.scc.bu.edu
cds-pub@scc-tc4.scc.bu.edu
cds-pub@scc-ga4.scc.bu.edu
cds-pub@scc-tb2.scc.bu.edu
cds-pub@scc-ga3.scc.bu.edu
cds-pub@scc-tc1.scc.bu.edu
cds-pub@scc-tc3.scc.bu.edu
cds-pub@scc-tc2.scc.bu.edu
cds-gpu-pub@scc-j11.scc.bu.edu
cds-gpu-pub@scc-305.scc.bu.edu
cds-gpu-pub@scc-306.scc.bu.edu
cds-gpu-pub@scc-j13.scc.bu.edu
cds-gpu-pub@scc-q32.scc.bu.edu
cds-gpu-pub@scc-j12.scc.bu.edu
cds-gpu-pub@scc-q31.scc.bu.edu
cds@scc-tb3.scc.bu.edu
cds@scc-tc4.scc.bu.edu
cds@scc-ga4.scc.bu.edu
cds@scc-tb2.scc.bu.edu
cds@scc-ga3.scc.bu.edu
cds@scc-tc1.scc.bu.edu
cds@scc-tc3.scc.bu.edu
cds@scc-tc2.scc.bu.edu
cds-gpu@scc-j11.scc.bu.edu
cds-gpu@scc-305.scc.bu.edu
cds-gpu@scc-306.scc.bu.edu
cds-gpu@scc-j13.scc.bu.edu
cds-gpu@scc-q32.scc.bu.edu
cds-gpu@scc-j12.scc.bu.edu
cds-gpu@scc-q31.scc.bu.edu
cds-m1024@scc-v08.scc.bu.eduAnd the count:
$ qselect -q *cds* | wc -l
32Of those which have at least 1 GPU and the count:
$ qselect -q *cds* -l gpus=1
cds-gpu-pub@scc-j11.scc.bu.edu
cds-gpu-pub@scc-305.scc.bu.edu
cds-gpu-pub@scc-306.scc.bu.edu
cds-gpu-pub@scc-j13.scc.bu.edu
cds-gpu-pub@scc-q32.scc.bu.edu
cds-gpu-pub@scc-j12.scc.bu.edu
cds-gpu-pub@scc-q31.scc.bu.edu
cds-gpu@scc-j11.scc.bu.edu
cds-gpu@scc-305.scc.bu.edu
cds-gpu@scc-306.scc.bu.edu
cds-gpu@scc-j13.scc.bu.edu
cds-gpu@scc-q32.scc.bu.edu
cds-gpu@scc-j12.scc.bu.edu
cds-gpu@scc-q31.scc.bu.edu
$ qselect -q *cds* -l gpus=1 | wc -l
14Example: CIFAR10 Training on CPU and GPU
Let’s put what we learned to use and train a CIFAR10 classifier with and without GPUs.
We’ll use the CIFAR10 classifer model training code from python environment notes but with some instrumentation for timing and GPU support.
It is important to note that you may be assigned 1 GPU on a multi-GPU node. You shouldn’t manually assign one of the GPUs.
Per this note you can check which GPU you are assigned with
import os
print(os.getenv("CUDA_VISIBLE_DEVICES"))We’ll modify our training code to check if CUDA is available and use it if so:
cifar-train.py
import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms
import time
# Record the start time
start_time = time.time()
# Check if CUDA is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using {device} device')
# Define transformations for the dataset
transform = transforms.Compose(
[transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
# Download and load the CIFAR-10 dataset
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=False)
# Classes
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
# Define the CNN model
class SmallCNN(nn.Module):
def __init__(self):
super(SmallCNN, self).__init__()
self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
self.pool = nn.MaxPool2d(2, 2)
self.fc1 = nn.Linear(32 * 8 * 8, 128)
self.fc2 = nn.Linear(128, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 32 * 8 * 8)
x = F.relu(self.fc1(x))
x = self.fc2(x)
return x
# Initialize the model, loss function, and optimizer
model = SmallCNN().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# Training the model
def train_model(model, trainloader, criterion, optimizer, epochs=5):
for epoch in range(epochs): # Loop over the dataset multiple times
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
# Get the inputs; data is a list of [inputs, labels]
inputs, labels = data[0].to(device), data[1].to(device)
# Zero the parameter gradients
optimizer.zero_grad()
# Forward pass
outputs = model(inputs)
loss = criterion(outputs, labels)
# Backward pass and optimize
loss.backward()
optimizer.step()
# Print statistics
running_loss += loss.item()
if i % 100 == 99: # Print every 100 mini-batches
print(f'[Epoch {epoch + 1}, Batch {i + 1}] loss: {running_loss / 100:.3f}')
running_loss = 0.0
print('Finished Training')
# Using the TorchScript method for model saving
# Important! Do not change the following 2 lines of code except for the model name
scripted_model = torch.jit.script(model)
torch.jit.save(scripted_model, 'cifar10-model.pt')
print('Model saved as cifar10-model.pt')
# Call the training function
train_model(model, trainloader, criterion, optimizer)
# Evaluation function to test the accuracy
def evaluate_model(model, testloader):
correct = 0
total = 0
with torch.no_grad():
for data in testloader:
images, labels = data[0].to(device), data[1].to(device)
outputs = model(images)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
accuracy = 100 * correct / total
print(f'Accuracy of the network on the 10,000 test images: {accuracy:.2f}%')
return accuracy
# Call the evaluation function
evaluate_model(model, testloader)
# Record the end time
end_time = time.time()
# Calculate the elapsed time
elapsed_time = end_time - start_time
minutes, seconds = divmod(elapsed_time, 60)
# Print the elapsed time in minutes and seconds
print(f"Elapsed time: {int(minutes)} minutes and {seconds:.2f} seconds")with the script
run-cifar-train.sh
#!/bin/bash -l
#$ -P dl4ds # Assign to project dl4ds
#$ -j y # merge the error and regular output into a single file
module load miniconda academic-ml/fall-2024
conda activate fall-2024-pyt
echo "Print python version"
python --version
python cifar-train.py
# to be processed correctly there must be a blank line at the end of the fileAnd submit both with and without GPU
$ qsub run-cifar-train.sh
Your job 317438 ("run-cifar-train.sh") has been submitted
$ qsub -l gpus=1 run-cifar-train.sh
Your job 317468 ("run-cifar-train.sh") has been submittedAnd let’s compare the outputs:
run-cifar-train.sh.o317438 (CPU)
Print python version
Python 3.11.9
Using cpu device
Files already downloaded and verified
Files already downloaded and verified
[Epoch 1, Batch 100] loss: 2.299
[Epoch 1, Batch 200] loss: 2.287
...
[Epoch 5, Batch 1400] loss: 1.113
[Epoch 5, Batch 1500] loss: 1.092
Finished Training
Model saved as cifar10-model.pt
Accuracy: 60.38%
Elapsed time: 2 minutes and 50.11 secondsrun-cifar-train.sh.o317468 (GPU)
Print python version
Python 3.11.9
Using cuda device
Files already downloaded and verified
Files already downloaded and verified
[Epoch 1, Batch 100] loss: 2.298
[Epoch 1, Batch 200] loss: 2.278
...
[Epoch 5, Batch 1400] loss: 1.116
[Epoch 5, Batch 1500] loss: 1.129
Finished Training
Model saved as cifar10-model.pt
Accuracy: 60.80%
Elapsed time: 0 minutes and 54.14 secondsReferences
- https://www.bu.edu/tech/support/research/system-usage/running-jobs/submitting-jobs/
- https://www.bu.edu/tech/support/research/system-usage/running-jobs/tracking-jobs/
- https://www.bu.edu/tech/support/research/software-and-programming/gpu-computing/
- https://www.bu.edu/tech/support/research/system-usage/running-jobs/batch-script-examples/
- https://www.bu.edu/tech/support/research/system-usage/running-jobs/process-reaper/