What is Snakemake & Why It Matters

01 The Pipeline Problem

Imagine you are processing 48 sorghum RNA-seq samples. Your pipeline has six steps: quality control with FastQC, trimming with Trimmomatic, alignment with HISAT2, sorting with samtools, counting with featureCounts, and differential expression with DESeq2. In a Bash script, you might write a for loop and chain every command together.

This works — until something breaks at step 4 on sample 23. Now you must re-run everything from the beginning, or carefully track exactly where it failed. If you add a new sample later, you run the whole script again even though most outputs already exist. If a collaborator wants to reproduce your results on a different cluster, they need your exact Bash script, your exact sample names, and hope nothing has changed.

The core issues with raw Bash pipelines are:

• No dependency tracking — Bash does not know that step 5 depends on step 4. If step 4 fails silently, step 5 runs on incomplete data.
• No resumability — If the pipeline crashes, you restart from zero unless you write complex checkpoint logic yourself.
• No parallelism — Running 48 samples sequentially wastes most of your compute time. Parallelising them with & and wait is fragile and hard to control.
• No reproducibility — The pipeline is tied to your exact machine, your exact paths, your exact software versions.

Snakemake was designed to solve all four of these problems at once.

02 What is Snakemake?

Snakemake is a Python-based workflow management system created by Johannes Köster at the University of Duisburg-Essen in 2012. It is now maintained by a large open-source community and is one of the two dominant workflow systems in bioinformatics (alongside Nextflow).

The key insight behind Snakemake is borrowed from GNU Make: instead of telling the computer how to run things step by step, you tell it what outputs you want and how to produce them from inputs. Snakemake figures out the rest — what order to run things in, which rules are already done, and how many rules can run simultaneously.

Snakemake is written in Python and its workflow files (Snakefile) use Python syntax. This means you can use any Python expression, import any Python library, and write arbitrarily complex logic inside a Snakemake pipeline — without leaving the Snakemake paradigm.

Why Snakemake for RNA-seq?

An RNA-seq pipeline is a perfect fit for Snakemake because it is inherently a directed acyclic graph (DAG) of file transformations. Each step takes files in and produces files out. Snakemake sees these file relationships and constructs the DAG automatically:

• FASTQ files → FastQC HTML reports (QC, no file transformation needed)
• FASTQ files → trimmed FASTQ files (Trimmomatic)
• Trimmed FASTQ → BAM files (HISAT2 alignment)
• BAM files → sorted BAM files (samtools sort)
• Sorted BAM → count matrix (featureCounts)
• Count matrix → differential expression results (DESeq2 in R)

03 How Snakemake Thinks

Understanding how Snakemake resolves rules is essential before writing any code. Snakemake works backwards from the target output.

When you ask Snakemake to produce results/counts.txt, it asks: "Which rule produces results/counts.txt?" It finds that rule, then looks at that rule's inputs. It asks: "Do these inputs already exist?" If not, it finds the rules that produce those inputs, and so on — building a dependency tree (DAG) from target back to source files.

Once the DAG is complete, Snakemake executes it in the correct order, running independent rules in parallel. If a rule has already produced its output and the output file is newer than the input, Snakemake skips that rule. This is called up-to-date checking — you get free resumability.

04 Installing Snakemake

Snakemake is best installed inside a dedicated conda environment. This keeps it isolated from other tools and makes it easy to recreate the environment on another machine.

# Create a dedicated environment for Snakemake
conda create -n snakemake-env -c conda-forge -c bioconda snakemake -y

# Activate the environment
conda activate snakemake-env

# Verify the installation
snakemake --version

Expected output from --version:

8.x.x

Create the project directory

Let's set up a clean project directory for this entire Snakemake module. We will build on this directory across all 10 lessons.

# Create the project tree for the entire module
mkdir -p ~/rnaseq-snakemake-pipeline/{data/raw,data/trimmed,results,logs,envs}

# Move into the project root
cd ~/rnaseq-snakemake-pipeline

# Confirm structure
tree -L 2

Expected output:

.
├── data
│   ├── raw
│   └── trimmed
├── envs
├── logs
└── results

05 Your First Rule

Every Snakemake workflow lives in a file called Snakefile (capital S, no extension). Let's write the simplest possible rule — one that copies a FASTQ file from data/raw/ to results/. This is intentionally trivial: the goal is to understand the anatomy of a rule before adding real bioinformatics commands.

First, create a small test FASTQ file to work with:

# Create a small fake FASTQ for testing (4-line FASTQ format)
cat > data/raw/sorghum_sample1.fastq << 'EOF'
@SRR1234567.1 sorghum read 1
ATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATCG
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
@SRR1234567.2 sorghum read 2
GCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTAGCTA
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
EOF

Now create the Snakefile:

# Snakefile — Lesson 1: your first rule
# Save this as: ~/rnaseq-snakemake-pipeline/Snakefile

# The 'rule all' tells Snakemake what the final target output is.
# It is always listed first and drives the entire pipeline.
rule all:
    input:
        "results/sorghum_sample1.fastq"


# This rule copies a FASTQ file from data/raw/ to results/
rule copy_fastq:
    input:
        "data/raw/sorghum_sample1.fastq"
    output:
        "results/sorghum_sample1.fastq"
    shell:
        "cp {input} {output}"

06 Dry Run & Execution

Before running any rule for real, always do a dry run first. A dry run tells you exactly what Snakemake would do — which rules it would execute and in what order — without actually executing any shell commands. This is one of Snakemake's most valuable features.

# Dry run — shows what would happen without running anything
snakemake --dry-run --printshellcmds

Expected dry-run output:

Building DAG of jobs...
Job stats:
job          count
-----------  -------
all          1
copy_fastq   1
total        2

Reasons:
    (check individual jobs above for details)
    output files have to be generated:
        copy_fastq: results/sorghum_sample1.fastq

rule copy_fastq:
    input: data/raw/sorghum_sample1.fastq
    output: results/sorghum_sample1.fastq
    jobid: 1
    reason: Missing output files: results/sorghum_sample1.fastq
    resources: tmpdir=/tmp

cp data/raw/sorghum_sample1.fastq results/sorghum_sample1.fastq

This was a dry-run (flag --dry-run). The order of jobs does not reflect the order of execution.

Now run it for real:

# Real run — executes all rules needed to produce the target
snakemake --cores 1

# Verify the output was created
ls -lh results/

Expected output from ls -lh results/:

total 4.0K
-rw-r--r-- 1 shajedur shajedur 200 Aug 27 10:43 sorghum_sample1.fastq

Run it again — watch Snakemake skip

Now run the exact same command a second time, without changing anything:

snakemake --cores 1

Expected output:

Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).

07 Quick Reference

08 Exercises

conda activate snakemake-env
snakemake --version
# Should print something like: 8.x.x

snakemake --help | head -40
# Look for: --jobs N or -j N (maximum number of parallel jobs)

rule all:
    input:
        "results/sorghum_sample1.fastq.gz"

rule copy_fastq:
    input:  "data/raw/sorghum_sample1.fastq"
    output: "results/sorghum_sample1.fastq"
    shell:  "cp {input} {output}"

rule compress_fastq:
    input:  "results/sorghum_sample1.fastq"
    output: "results/sorghum_sample1.fastq.gz"
    shell:  "gzip -k {input}"

conda install -c conda-forge graphviz -y
snakemake --dag | dot -Tsvg > dag.svg
xdg-open dag.svg   # Linux viewer