What is HPC? Clusters, Nodes & the Bioinformatics Case

01 Why bioinformatics needs HPC

A modern whole-genome sequencing run produces roughly 100 GB of raw FASTQ data per sample. A rice population study might have 200 accessions. A sorghum GWAS panel — like the one behind your BPV genomic selection thesis — may contain tens of thousands of SNP markers across hundreds of lines. Your laptop has 8–16 GB of RAM and 4–8 CPU cores. The maths does not work.

HPC stands for High-Performance Computing. It refers to a networked collection of computers (called a cluster) that work together and are managed by a job scheduler. Instead of running your STAR alignment on your laptop, you write a script that says "I need 16 CPUs, 64 GB RAM, and 4 hours" — and the scheduler finds the machines and runs your job automatically, even while you sleep.

Here is how computational demand scales for common bioinformatics tasks:

02 Anatomy of a cluster

A computing cluster is not a single computer. It is a collection of many computers — called nodes — connected by a high-speed internal network (InfiniBand or 10 GbE). Each node looks like a powerful server with many CPU cores, large RAM, and fast local storage. The nodes share access to a common, large parallel filesystem (like GPFS or Lustre) where your data and scripts live.

Key components you need to understand:

03 Login nodes vs compute nodes

This is the most important rule of HPC etiquette: never run heavy computation on the login node.

What is allowed on the login node vs what requires a job submission:

04 Where SLURM fits in

SLURM stands for Simple Linux Utility for Resource Management. It is the job scheduler used by the vast majority of HPC clusters worldwide — including those at German universities (HRZ, HLRS, NHR centres). Its job is simple in concept: accept jobs from hundreds of users, decide who gets what resources and when, and execute them in order.

SLURM's core commands — you will learn every one of these in depth across this module:

A SLURM job script is just a Bash script with special comment lines at the top that tell SLURM what resources your job needs. Here is what a minimal one looks like — we will build real ones from Lesson 4 onwards:

#!/bin/bash
#SBATCH --job-name=star_align        # name shown in squeue
#SBATCH --ntasks=1                   # number of tasks
#SBATCH --cpus-per-task=8            # CPU cores per task
#SBATCH --mem=64G                    # RAM
#SBATCH --time=02:00:00              # max wall-clock time (HH:MM:SS)
#SBATCH --output=logs/star_%j.out   # stdout log (%j = job ID)
#SBATCH --error=logs/star_%j.err    # stderr log

# Load the STAR module
module load STAR/2.7.10a

# Run the alignment
STAR --runThreadN 8 \
     --genomeDir /scratch/shajedur/sorghum_genome/star_index \
     --readFilesIn sample_R1.fastq.gz sample_R2.fastq.gz \
     --outSAMtype BAM SortedByCoordinate \
     --outFileNamePrefix results/sample_

05 A real bioinformatics workflow on HPC

To make this concrete, here is how a typical RNA-seq analysis looks when run on an HPC cluster. You would run this for your sorghum RNA-seq data — each step becomes a separate SLURM job, and they chain together using job dependencies.

Notice you only interact with the cluster for job submission and data transfer. All the heavy computation happens automatically on compute nodes while you do other work. You can log off completely — your jobs keep running.

Key vocabulary summary

06 Exercises

No cluster access is required for these exercises — they are conceptual and research tasks to solidify your understanding before you SSH into a real cluster in Lesson 2.

# 50,000 SNPs × 300 lines × 8 bytes per value
50000 × 300 × 8 = 120,000,000 bytes = ~115 MB

# With 200,000 SNPs
200000 × 300 × 8 = 480,000,000 bytes = ~457 MB

# But rrBLUP builds a kinship matrix (G matrix): n × n
# 300 × 300 × 8 = 720,000 bytes — tiny
# BGLR with MCMC chains needs more: 2–4 GB for large panels
# Request 16–32 GB on HPC to be safe