Why Plan Before You Code
Every professional genomics pipeline starts on paper, not on the terminal. Before you run a single command, you need to know: Where does my data come from? Where does it go? What question am I answering? Bioinformatics pipelines can run for hours or days — a planning mistake discovered at step 8 means re-running steps 1 through 7. Good design saves hundreds of compute hours.
In this capstone module you will build a real, reproducible plant genomics pipeline for Sorghum bicolor — the same species used in Shajedur's own thesis research. The pipeline integrates every skill you have built across the previous 22 modules: Bash, Git, Conda, R, Python, QC, alignment, variant calling, differential expression, and Snakemake automation.
This first lesson is a design lesson. You will produce three deliverables before Lesson 2:
- A written pipeline overview (what each step does and why)
- A directory structure created on your Ubuntu system
- A Conda environment with all required tools installed and verified
The 10-Step Pipeline Overview
Here is the full pipeline you will build across this module. Each lesson corresponds to one stage. Read every step carefully — understanding why each stage exists is as important as knowing how to run it.
Why Sorghum bicolor? Sorghum is a C4 cereal crop of global importance, especially for drought-prone regions. Its reference genome (BTx623 v3.1) is well-annotated and freely available from Phytozome and NCBI. It is also the species used in Shajedur's own thesis — making this pipeline directly applicable to real research.
Choosing Your Dataset
A capstone pipeline needs real data, not toy files. But downloading 100 GB of raw sequencing data overnight is not practical. The strategy here is to use a small, well-characterised public RNA-seq dataset from SRA that has biological replicates (required for DESeq2) and an interesting biological question — drought stress or nitrogen response in Sorghum — so your results are scientifically meaningful, not random numbers.
We will use publicly available Sorghum bicolor RNA-seq data from NCBI SRA. A good candidate
dataset is PRJNA396116 — drought stress in Sorghum, with 6 samples (3 control, 3
drought), paired-end 150 bp reads, approximately 3–5 GB per sample. You will download these in
Lesson 2 using prefetch and
fasterq-dump from SRA-tools.
Storage requirement: Plan for at least 50–80 GB of free disk space on your Ubuntu drive. Raw FASTQ files are large. We will delete intermediates as we go to keep disk usage manageable. If space is limited, use only 2 samples per group (2 control + 2 drought) — DESeq2 requires a minimum of 2 replicates per condition.
The reference genome for this pipeline is Sorghum bicolor BTx623 v3.1, available from Phytozome (JGI). It is approximately 730 Mb as a compressed FASTA. The GTF annotation file (needed for alignment and featureCounts) is included with the genome download. In Lesson 4 you will download both and build the alignment indexes.
Setting Up the Directory Structure
Professional bioinformatics pipelines always use a predictable, hierarchical directory layout. This is not just tidiness — it is what allows Snakemake (and your future self) to find every file by rule. If you start with a messy folder, your pipeline will become unmaintainable. Set up the structure once, correctly, on day one.
Here is the full directory layout for the capstone project. Create it now using the mkdir -p command below.
# Create the full project directory tree in one command mkdir -p ~/plant-genomics-pipeline/{data/{raw,trimmed,reference},indexes/{star,hisat2},results/{qc,alignment,variants,counts,de_analysis,plots},scripts/{bash,r,python},workflow,envs,logs} # Move into the project root cd ~/plant-genomics-pipeline # Verify the structure was created correctly find . -type d | sort ./data ./data/raw ./data/trimmed ./data/reference ./indexes ./indexes/star ./indexes/hisat2 ./results ./results/qc ./results/alignment ./results/variants ./results/counts ./results/de_analysis ./results/plots ./scripts ./scripts/bash ./scripts/r ./scripts/python ./workflow ./envs ./logs
The raw/ directory is sacred. Never modify or delete any file in data/raw/. Always work on copies or outputs. This ensures you can re-run any step from scratch without re-downloading data, which saves hours.
Next, create a README.md that documents the project from day one.
cat > README.md << 'EOF' # Plant Genomics Pipeline — Sorghum bicolor A complete end-to-end plant genomics analysis pipeline for Sorghum bicolor RNA-seq and variant calling data. ## Biological Question How does drought stress alter gene expression and genomic variation in Sorghum bicolor (BTx623)? ## Dataset - **Source:** NCBI SRA — BioProject PRJNA396116 - **Species:** Sorghum bicolor BTx623 - **Reference:** v3.1 genome from Phytozome - **Design:** 3 control vs 3 drought-stressed samples (PE 150 bp) ## Pipeline Steps 1. Data acquisition and QC 2. Read trimming (fastp) 3. Reference genome indexing (STAR) 4. Alignment (STAR) 5. Variant calling (GATK4 HaplotypeCaller) 6. Variant filtering and annotation (SnpEff) 7. Differential expression (DESeq2) 8. Visualisation (ggplot2 + Python) 9. Full Snakemake automation ## Author Shajedur Rahman Hossain MSc Agrobiotechnology — JLU Gießen, Germany GitHub: https://github.com/shajedurhossain EOF # Confirm it was written cat README.md
Tool Inventory
Before installing anything, document exactly which tools you need and why. This table is your pipeline's dependency manifest — the same information Snakemake will use in Lesson 10.
Creating the Conda Environment
All 12 tools in this pipeline must work together. Conda environments isolate tool versions so that, for example, GATK's Java requirements do not conflict with SnpEff's. Creating the environment from a YAML file also means anyone on any Ubuntu machine can reproduce your exact pipeline with a single command — this is what makes bioinformatics reproducible science.
First, write the environment YAML file. This file declares every tool and its channel, so Conda can resolve all version conflicts automatically.
# Write the Conda environment specification cat > envs/plant-genomics.yaml << 'EOF' name: plant-genomics channels: - bioconda - conda-forge - defaults dependencies: - python=3.11 - sra-tools=3.0 - fastqc=0.12 - multiqc=1.19 - fastp=0.23 - star=2.7 - hisat2=2.2 - samtools=1.19 - gatk4=4.4 - snpeff=5.1 - subread=2.0 - snakemake-minimal=7.32 - r-base=4.3 - bioconductor-deseq2 - bioconductor-edger - r-ggplot2 - r-pheatmap - r-ggrepel - pip - pip: - multiqc EOF envs/plant-genomics.yaml written
Now create the environment. This step will take 10–20 minutes the first time as Conda downloads and resolves all packages.
# Create the environment from the YAML file conda env create -f envs/plant-genomics.yaml # Activate it conda activate plant-genomics # Verify key tools are installed and accessible echo "=== Checking tool versions ===" fastqc --version fastp --version STAR --version samtools --version | head -1 gatk --version snakemake --version === Checking tool versions === FastQC v0.12.1 fastp 0.23.4 2.7.10a samtools 1.19 4.4.0.0 7.32.4
GATK4 alternative: If gatk4 fails to install via Conda (it sometimes conflicts with Java versions), install it manually: download the GATK4 JAR from github.com/broadinstitute/gatk/releases and call it with java -jar gatk.jar. Lesson 6 will cover this.
Finally, initialise the Git repository so every change is tracked from lesson one.
# Initialise Git and make the first commit git init # Create a .gitignore to exclude large data files cat > .gitignore << 'EOF' # Large data files — do not commit to GitHub data/raw/ data/trimmed/ data/reference/ indexes/ results/alignment/*.bam results/alignment/*.bai results/variants/*.vcf.gz logs/ # Keep directory structure !data/raw/.gitkeep !data/trimmed/.gitkeep !data/reference/.gitkeep !indexes/.gitkeep EOF # Add .gitkeep files so empty directories are tracked touch data/raw/.gitkeep data/trimmed/.gitkeep data/reference/.gitkeep indexes/.gitkeep # Stage and commit git add . git commit -m "lesson 1: project design — directory structure and conda env" [main (root-commit) a1b2c3d] lesson 1: project design — directory structure and conda env 8 files changed, 74 insertions(+)
Never commit BAM or FASTQ files to GitHub. They are too large (often gigabytes each) and GitHub has a 100 MB file limit. The .gitignore above handles this automatically. Only scripts, YAML configs, and results (tables, plots) should be committed.
Exercises
In your own words (2–3 sentences), write down the biological question this pipeline will answer. Include: the organism, the experimental condition, and what type of genomic data you are analysing. Write this into a file called QUESTION.md in the project root.
Show answer
To write it:
nano ~/plant-genomics-pipeline/QUESTION.md
Before Lesson 2 you will download several gigabytes of FASTQ data. Check how much free space is available on your Ubuntu partition and confirm you have at least 50 GB free. Use the df command.
Show answer
df -h ~
# Look for the "Avail" column on the row for your home partition
# You need at least 50G available
Create a new GitHub repository called plant-genomics-pipeline on github.com/shajedurhossain and connect your local repository to it. Push the initial commit you made in Section 6.
Show answer
cd ~/plant-genomics-pipeline
git remote add origin https://shajedurhossain:TOKEN@github.com/shajedurhossain/plant-genomics-pipeline.git
git branch -M main
git push -u origin main
Replace
TOKEN with your Personal Access Token (classic, repo scope).
🎯 You have designed your complete plant genomics pipeline.