Osman Goni Ridwan

VASP cheat sheet · Essentials

VASP Slurm job script cheat sheet

How to run VASP under Slurm: CPU and GPU job scripts, how ranks, threads and the parallel tags fit together, chains of jobs, stopping cleanly before the time limit, and watching jobs. The generator writes a script and the matching INCAR tags.

Checked against VASP 6.6.1 · Updated

The short version

  1. Pick the executable: vasp_std for normal runs, vasp_gam for a single Γ point, vasp_ncl for noncollinear magnetism and spin–orbit coupling.
  2. Set ulimit -s unlimited and OMP_NUM_THREADS in every script; most random segmentation faults are a stack that is too small.
  3. Launch with the MPI that VASP was built with: srun where the site supports it, never a mpirun from another MPI.
  4. KPAR must divide the number of MPI ranks; NCORE should divide the ranks per node. Set NCORE or NPAR, never both.
  5. On GPUs: one MPI rank per GPU, no NCORE (VASP forces 1), KPAR = number of GPUs if memory allows, a larger NSIM.
  6. Let Slurm warn the script before the time limit and write a STOPCAR, so VASP stops cleanly after an ionic step.

Write a job script

Describe the layout and the generator writes a Slurm script and the parallel tags for the INCAR. The script is generic: replace ACCOUNT, PARTITION and the module name with your site's values. It runs in your browser.

Templates

CPU job (MPI, one thread per rank)
#!/bin/bash
# Replace ACCOUNT, PARTITION and the module name with your site's values.
#SBATCH -J relax
#SBATCH -A ACCOUNT
#SBATCH -p PARTITION
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH -c 1
#SBATCH -t 24:00:00
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

module purge
module load vasp/VERSION

ulimit -s unlimited                   # VASP keeps large arrays on the stack
export OMP_NUM_THREADS=1
export OMP_STACKSIZE=512m
export OMP_PLACES=cores OMP_PROC_BIND=close

cd "$SLURM_SUBMIT_DIR" || exit 1
srun -n 128 -c 1 --cpu-bind=cores vasp_std > vasp.out
GPU job (OpenACC build): one MPI rank per GPU
#!/bin/bash
# Replace ACCOUNT, GPU_PARTITION and the module name with your site's values.
#SBATCH -J relax-gpu
#SBATCH -A ACCOUNT
#SBATCH -p GPU_PARTITION
#SBATCH -N 1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks-per-node=4
#SBATCH -c 16
#SBATCH -t 12:00:00
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

module purge
module load vasp/VERSION-gpu

ulimit -s unlimited
export OMP_NUM_THREADS=16             # CPU-side threads per rank
export OMP_STACKSIZE=512m
export OMP_PLACES=cores OMP_PROC_BIND=close

cd "$SLURM_SUBMIT_DIR" || exit 1
srun -n 4 -c 16 --cpu-bind=cores --gpu-bind=none vasp_std > vasp.out

INCAR for the GPU job: no NCORE, KPAR = 4 if the irreducible k-points divide evenly and memory allows, and NSIM well above 4 (benchmark 8 to 32).

Ranks, threads and the parallel tags

SettingRule
Cores usednodes × --ntasks-per-node × -c. Keep it at or below the physical cores: oversubscription slows everything.
-c and threads-c equals OMP_NUM_THREADS (twice that if the nodes count hyperthreads as CPUs). Pass -c to srun as well as to sbatch.
KPARDivides the MPI ranks and, ideally, the number of irreducible k-points (on GPUs, only when they divide evenly). Each k-point group holds its own copy of the data, so it costs memory. Small bulk cells with many k-points: large KPAR, NCORE = 1. VASP advice
NCORERanks working on one band. A divisor of the ranks per node, near √(ranks per k-point group); the cores per NUMA domain is often best. About 4 for ~100 atoms, 12–16 above 400 atoms. Forced to 1 with OpenMP or on GPUs. VASP advice
NPARThe old way to set the same thing (NPAR × NCORE = ranks per k-point group). If both are set, NPAR wins: delete it from old templates.
IMAGES (NEB)The ranks are split over the images too: total ranks = IMAGES × KPAR × (ranks per k-point group).

VASP's own recipe: do a dry run (vasp_std --dry-run, VASP 6.4 and later) to get NBANDS and the number of irreducible k-points, benchmark a few NCORE and KPAR values on a few SCF steps of the real system, then keep the fastest. The layout VASP actually used is at the top of OUTCAR.

TagWhat it doesDefaultWhat to useWatch out
NCORENumber of cores that work together on one band.1Cores per NUMA domain is a good start; about 4 for ~100 atoms, 12–16 above 400 atoms. Benchmark.Forced to 1 with OpenMP threads and on GPUs. Do not set NPAR as well.
NPARLegacy: the number of band groups (MPI ranks ÷ NCORE).the number of MPI ranksUse NCORE instead.If both are set, NPAR wins.
KPARNumber of k-point groups that run in parallel.1A divisor of the MPI ranks, up to the number of irreducible k-points if memory allows; on GPUs often the number of GPUs.Each group holds its own copy of the data: memory is not shared.
NSIMNumber of bands optimized at once in the blocked algorithms.4Higher on GPUs (up to 32).

Which executable

ExecutableUse it for
vasp_stdEvery collinear run, spin-polarized or not, with a k-point mesh. The default.
vasp_gamRuns with only the Γ point: molecules, large supercells, most AIMD.
vasp_nclNoncollinear magnetism and spin–orbit coupling (LNONCOLLINEAR, LSORBIT). vasp_std stops on such an INCAR.

Many structures and chained jobs

Job array: one directory per task (dirs.txt lists them)
#!/bin/bash
# ... the #SBATCH lines of a template, plus:
#SBATCH -a 0-199%20
#SBATCH -o logs/%x-%A_%a.out
DIR=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" dirs.txt)
cd "$DIR" || exit 1
srun vasp_std > vasp.out

0-199%20 runs 200 tasks, at most 20 at a time. Create the log folder first (mkdir -p logs): Slurm does not, and tasks without it fail without a log.

Relax, then static, then DOS
jid1=$(sbatch --parsable relax.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 static.sh)
sbatch --dependency=afterok:$jid2 dos.sh

afterok checks only the exit code, and VASP exits normally after NSW steps even when the relaxation did not converge. Start the next script with a guard:

Guard at the top of static.sh
grep -q "reached required accuracy" ../relax/OUTCAR || { echo "relaxation not converged"; exit 1; }
cp ../relax/CONTCAR POSCAR

Stop cleanly before the time limit

A job killed at the time limit can leave a half-written WAVECAR. A STOPCAR with LSTOP = .TRUE. stops VASP after the current ionic step; LABORT = .TRUE. stops at the next electronic step, leaving unconverged WAVECAR and CHGCAR, so prefer LSTOP. Slurm can send the script a signal a set time before the limit; the script then writes the STOPCAR.

Signal trap: replace the srun line of a template with this
#!/bin/bash
#SBATCH --signal=B:USR1@900
# ... the other #SBATCH lines; leave enough time (here 15 min) for about three ionic steps
rm -f STOPCAR                         # a leftover STOPCAR would stop the run at once
trap 'echo "LSTOP = .TRUE." > STOPCAR' USR1
srun vasp_std > vasp.out &
pid=$!
wait $pid; status=$?
if [ $status -gt 128 ]; then          # the signal interrupted wait: let VASP finish its step
  wait $pid; status=$?
fi
exit $status                          # keeps the exit code for --dependency=afterok

Size the warning from the time per ionic step: grep LOOP+ OUTCAR prints it. To continue: keep OUTCAR, OSZICAR and vasprun.xml under new names, cp CONTCAR POSCAR, keep WAVECAR and CHGCAR (and TAUCAR for meta-GGA), resubmit.

Watch and manage jobs

Slurm commands
squeue -u $USER                          # your jobs; the REASON column explains pending ones
scontrol show job JOBID                  # the full record: time limit, nodes, work directory
sacct -j JOBID --format=JobID,JobName%20,State,ExitCode,Elapsed,Timelimit,NNodes,NTasks,MaxRSS
seff JOBID                               # CPU and memory efficiency after the job ends (if installed)
scancel JOBID                            # cancel one job; scancel -u $USER cancels all yours
sinfo -s                                 # partitions and how many nodes are free

In sacct, State TIMEOUT means the time limit, OUT_OF_MEMORY an out-of-memory kill, NODE_FAIL a node problem. MaxRSS is recorded on job steps, so leave out -X when you want memory. Progress of a running job: tail -f vasp.out, or grep -c " F= " OSZICAR for the ionic steps done.

Common mistakes

  • No ulimit -s unlimited: random segmentation faults.
  • A mpirun from a different MPI than VASP's: one rank with many threads, or a crash.
  • Ranks × threads above the physical cores, or -c different from OMP_NUM_THREADS.
  • KPAR that does not divide the ranks; NCORE and NPAR both set.
  • More than one MPI rank per GPU, or NCORE set on GPUs.
  • Trusting afterok as proof of convergence; a stale STOPCAR in the directory.
  • Several calculations in one directory, or running in a home directory instead of scratch.
  • Spin-polarized r²SCAN on VASP 6.6.0, whose forces are wrong: check head -1 OUTCAR.
  • With KPAR > 1 a run can hang in the SCF (a known issue, partly fixed in 6.6.0): if a job stalls, try KPAR = 1.

The physics behind this sheet

  • Lecture 19: VASP input files and convergence tests (in preparation)

Further reading

  1. VASP Wiki, “Optimizing the parallelization” (NCORE, KPAR, the dry run and benchmarking). · www.vasp.at
  2. VASP Wiki, “GPU ports of VASP” (one MPI rank per GPU, NCORE, KPAR and NSIM on GPUs). · www.vasp.at
  3. VASP Wiki, “Combining MPI and OpenMP” (threads, stack size and pinning). · www.vasp.at
  4. Slurm documentation, “sbatch” (directives, --signal, --dependency and arrays). · slurm.schedmd.com