Skip to content

Differential Gene Expression Analysis in R with DESeq2

Oak
A feathered reptilian creature on a dark studio pedestal, its plumage cool blue on one side and warm amber on the other, with glowing spine filaments.

By the end of this guide you will have a table of genes ranked by shrunken log2 fold change with adjusted p-values, a variance-stabilized expression matrix suitable for clustering and principal component analysis, and a set of diagnostic plots that tell you whether the model fit is trustworthy. You need R 4.3 or later, plus per-sample transcript quantifications from Salmon or a gene-level count matrix from featureCounts. The Bioconductor packages you need are listed below.

  • DESeq2
  • tximport
  • apeglm
  • AnnotationDbi
  • org.Hs.eg.db
  • fgsea

You also need a sample sheet with one row per library and at least one column describing the comparison you care about. Roughly 16 GB of RAM is enough for a few dozen human libraries. The memory cost comes from the count matrix and the model fit. The reads do not drive it.

A note on scope before the code. Differential expression is a measurement of relative RNA abundance between groups of samples. It does not identify causes and it does not make clinical determinations. Changes in transcript abundance in tissue can arise from several distinct upstream mechanisms. Whole-genome studies of tumors have shown this when they trace expression shifts back to point mutations, structural variants, and copy-number changes in the same pathway.1 Interpretation of any result that bears on your health belongs with a clinician.

1. Quantify transcripts before you open R

Do the alignment-free quantification step outside R and only once. We recommend Salmon with a decoy-aware index built from the GENCODE transcriptome plus the primary-assembly genome as decoy sequence. The decoys reduce spurious assignment of reads that originate outside the annotated transcriptome.

# Build the index once (GENCODE v44 human, for example)
grep '^>' <(gunzip -c GRCh38.primary_assembly.genome.fa.gz) \
  | cut -d ' ' -f 1 | sed 's/>//g' > decoys.txt
cat gencode.v44.transcripts.fa.gz GRCh38.primary_assembly.genome.fa.gz > gentrome.fa.gz
salmon index -t gentrome.fa.gz -d decoys.txt -p 12 \
  -i salmon_index_v44 --gencode -k 31

Then quantify each library. The -l A flag lets Salmon infer library type, which is safer than guessing strandedness by hand.

salmon quant -i salmon_index_v44 -l A \
  -1 sample01_R1.fastq.gz -2 sample01_R2.fastq.gz \
  -p 12 --gcBias --seqBias --posBias \
  --numGibbsSamples 20 --validateMappings \
  -o quants/sample01

Check the mapping rate in quants/sample01/logs/salmon_quant.log. For polyA-selected human libraries against a GENCODE index we expect 75 to 90 percent. Below 60 percent, stop and investigate: the usual causes are adapter contamination, the wrong species index, or heavy rRNA and globin carryover. --numGibbsSamples 20 writes inferential replicates that let you distinguish genes with genuinely uncertain quantification, which matters for multi-isoform genes. The human annotation is dense with alternative isoforms, and ENCODE’s transcript catalog is the reason a gene-level count is a sum over many competing assignments rather than a direct read tally.2

If you have gene-level counts from featureCounts instead, that is fine. Use -p --countReadPairs for paired-end data and set -s explicitly to 0, 1, or 2 based on the library protocol. Getting -s wrong silently throws away most of your reads.

2. Import quantifications into R at the gene level

tximport aggregates transcript estimates to genes and produces an offset that corrects for average transcript length per sample. That offset matters. Two samples can differ in apparent gene-level counts purely because isoform usage shifted the effective length.

library(tximport)
library(DESeq2)

samples <- read.csv("samples.csv", stringsAsFactors = FALSE)
# samples: id, condition, batch, rin, sex
files <- file.path("quants", samples$id, "quant.sf")
names(files) <- samples$id
stopifnot(all(file.exists(files)))

# tx2gene: two columns, transcript ID then gene ID
tx2gene <- read.delim("gencode.v44.tx2gene.tsv", header = FALSE,
                      col.names = c("tx", "gene"))

txi <- tximport(files, type = "salmon", tx2gene = tx2gene,
                ignoreTxVersion = FALSE)

dds <- DESeqDataSetFromTximport(txi, colData = samples,
                                design = ~ batch + condition)

Set factor levels explicitly rather than relying on alphabetical order. DESeq2 reports fold changes relative to the first level, and silently comparing in the wrong direction is one of the most common errors in this workflow.

dds$condition <- factor(dds$condition, levels = c("control", "treated"))
dds$batch <- factor(dds$batch)

3. Filter low-count genes

Filtering is about power. It is not about correctness. Genes with almost no reads carry no information and inflate the multiple testing burden. We keep genes with at least 10 counts in at least as many samples as the smallest group.

smallest_group <- min(table(dds$condition))
keep <- rowSums(counts(dds) >= 10) >= smallest_group
dds <- dds[keep, ]
nrow(dds)   # expect roughly 14,000-18,000 for bulk human tissue

If you are left with fewer than 10,000 genes, the libraries are probably shallow or degraded. If you are left with more than 25,000, check that you are not counting every annotated lncRNA and pseudogene in a low-complexity library.

4. Inspect structure with variance-stabilized data

Do this before fitting the model you care about. Exploratory analysis on the wrong scale is misleading, because raw counts have variance that grows with the mean. Use vst for more than about 20 samples and rlog for small sets.

vsd <- vst(dds, blind = TRUE)
plotPCA(vsd, intgroup = c("condition", "batch"))

# Sample-sample distances
library(pheatmap)
d <- dist(t(assay(vsd)))
pheatmap(as.matrix(d), clustering_distance_rows = d,
         clustering_distance_cols = d)

Read the PCA before committing to a design. If PC1 separates by batch or by RNA integrity number rather than by condition, your effect of interest is smaller than your nuisance variation. That is not failure. The result is information. It tells you which covariates belong in the design formula.

Two more checks. Correlate PC1 and PC2 against continuous covariates such as RIN, library size, and duplication rate. And confirm sex using expression of XIST and RPS4Y1, which catches sample swaps faster than any other single test.

cor(vsd$rin, prcomp(t(assay(vsd)))$x[, 1:3])

5. Fit the model and check dispersion

The DESeq2 model is a negative binomial generalized linear model per gene, with dispersion estimates shrunk toward a fitted mean-dispersion trend. One call does size factor estimation, dispersion estimation, and fitting.

dds <- DESeq(dds)
plotDispEsts(dds)

Inspect that plot. The black points are gene-wise dispersion estimates, the red line is the fitted trend, and the blue points are final shrunken values. You want the trend decreasing with mean expression and most genes pulled toward it. Two failure signatures matter. A cloud of high-dispersion outliers that resist shrinkage usually means unmodeled sample heterogeneity. A trend line that is flat and high across the whole expression range usually means your replicates are not replicates.

Also check the size factors. A range wider than roughly fivefold suggests very uneven library depth or a few samples dominated by a handful of transcripts.

sizeFactors(dds)

6. Extract results with shrunken fold changes

Never rank genes by raw log2 fold change. Low-count genes produce extreme and unreliable estimates. lfcShrink with the apeglm estimator gives a posterior estimate that is usable for ranking and for plotting.

resultsNames(dds)
# [1] "Intercept" "batch_2_vs_1" "condition_treated_vs_control"

res <- results(dds, name = "condition_treated_vs_control", alpha = 0.05)
res_shrunk <- lfcShrink(dds, coef = "condition_treated_vs_control",
                        type = "apeglm")

summary(res)
plotMA(res_shrunk, ylim = c(-4, 4))

If you care about effect size rather than any nonzero difference, test against a threshold directly instead of filtering the output by fold change. Post-hoc filtering on fold change breaks the false discovery rate control that padj provides.

res_thr <- results(dds, name = "condition_treated_vs_control",
                   lfcThreshold = log2(1.5), altHypothesis = "greaterAbs")

Then annotate. Strip GENCODE version suffixes first, or every mapping will return NA.

library(AnnotationDbi); library(org.Hs.eg.db)

tab <- as.data.frame(res_shrunk)
tab$ensembl <- sub("\\..*$", "", rownames(tab))
tab$symbol <- mapIds(org.Hs.eg.db, keys = tab$ensembl,
                     column = "SYMBOL", keytype = "ENSEMBL",
                     multiVals = "first")
tab <- tab[order(tab$padj), ]
write.csv(tab, "de_condition_treated_vs_control.csv")

7. Move from genes to pathways

Individual gene hits are noisy and hard to interpret. Aggregate statistics across gene sets are more stable, because coordinated modest changes in many genes of one pathway are more informative than one large change in one gene. Studies of diseased human tissue have made this point repeatedly. Expression profiling of coronary arteries in ischemic heart disease pointed toward an inflammatory program as a set. It did not point toward a single gene.3

Use fgsea on a ranking statistic rather than a hypergeometric test on a thresholded list. The threshold-free approach uses the whole ranking and avoids an arbitrary cutoff.

library(fgsea)

stat <- tab$stat
names(stat) <- tab$symbol
stat <- stat[!is.na(names(stat)) & !is.na(stat)]
stat <- stat[!duplicated(names(stat))]

pathways <- gmtPathways("c2.cp.reactome.v2023.2.Hs.symbols.gmt")
fg <- fgsea(pathways, stat, minSize = 15, maxSize = 500, nPermSimple = 10000)
head(fg[order(fg$padj), c("pathway", "padj", "NES", "size")], 20)

Use the Wald statistic from results, not the shrunken fold change, as the ranking metric. The statistic already incorporates uncertainty. The shrunken fold change is for display and for effect-size reasoning.

8. Remove nuisance variation for visualization only

When batch is in the design formula, DESeq2 already accounts for it in the test. Plots built from vst output do not know that. To see condition separation clearly, subtract the batch term from the transformed matrix before plotting.

library(limma)
mat <- assay(vsd)
mat <- limma::removeBatchEffect(mat, batch = vsd$batch,
                                design = model.matrix(~ vsd$condition))
assay(vsd) <- mat
plotPCA(vsd, intgroup = "condition")

Do not feed a batch-corrected matrix back into DESeq(). The model expects raw integer counts and handles covariates through the design.

If you suspect hidden structure that no recorded covariate captures, estimate surrogate variables with sva and add them to the design. This is the right tool when a large fraction of genes shifts in unison for reasons you cannot name.

9. Sanity-check the result before you believe it

Three checks catch most false confidence. First, plot the individual counts for your top hits and confirm the pattern is driven by the whole group rather than one sample.

plotCounts(dds, gene = rownames(tab)[1], intgroup = "condition", normalized = TRUE)

Second, look at the p-value histogram. Under the null it should be flat with a spike near zero. A histogram that rises toward one points at a misspecified model or dependent samples. A U-shape usually means the variance is being overestimated.

hist(res$pvalue[res$baseMean > 10], breaks = 50)

Third, run a permuted-label negative control. Shuffle the condition labels within batch, refit, and count hits at padj < 0.05. A well-specified model with no real signal should return close to zero genes. If permuted labels give hundreds of hits, the design is wrong and every result from the real fit is suspect.

10. Cross-species and orthology, if that is where your hypothesis came from

A large fraction of candidate genes come from mouse work, so you will often need to map between species. Use a curated orthology resource rather than symbol matching, because symbol conventions diverge and one-to-many relationships are common. Comparative analysis of the mouse and human genomes established both the broad conservation of protein-coding content and the substantial number of lineage-specific gene family expansions that break naive one-to-one mapping.4

library(babelgene)
orth <- orthologs(genes = c("Acta2", "Il6", "Cd68"), species = "mouse", human = FALSE)

Treat any mouse-derived direction of effect as a hypothesis to test in human data. It is not a prediction to confirm.

Common problems

Fold changes point the wrong way. This comes from factor level order. Check resultsNames(dds) and read the coefficient name literally: condition_treated_vs_control means treated relative to control.

Nearly all annotation lookups return NA. GENCODE gene IDs carry a version suffix such as ENSG00000141510.16, and org.Hs.eg.db keys do not. Strip with sub("\\..*$", "", id). For PAR_Y genes, drop the _PAR_Y suffix rows entirely or they will collide with their X-linked counterparts.

Almost no genes reach significance. The three usual causes are low replication, a large batch effect in the design, and heterogeneous samples. Run the permutation check first to confirm the model is not simply underpowered rather than broken. With fewer than three samples per group, DESeq2 will run but the dispersion estimates rest on very little information.

Thousands of genes are significant with tiny fold changes. This is typical of a strong confounder or a global shift in RNA composition, for example heavy globin or rRNA content in a subset of samples. Check the p-value histogram and the correlation of PC1 with library composition metrics.

Independent filtering removed genes you expected to see. results() applies filtering on mean expression to maximize discoveries, which sets padj to NA for low-mean genes. Inspect with metadata(res)$filterThreshold and turn it off with independentFiltering = FALSE if you need p-values for every tested gene.

Counts and metadata are misaligned. DESeqDataSetFromTximport does not reorder for you. Always assert identical(colnames(txi$counts), samples$id) before constructing the object.

The comparison is one timepoint versus another in a single person. This is a paired design with n of 1 per condition, and standard group-level dispersion estimation does not apply. Longitudinal single-subject work needs repeated sampling across many timepoints and a model that treats within-person variance explicitly. Do not treat a single-person expression difference as evidence of a health condition. Making that determination requires a clinician.

Footnotes

  1. Phuc H. Hoang, Sara E. Dobbins, Alex J. Cornish, et al. Whole-genome sequencing of multiple myeloma reveals oncogenic pathways are targeted somatically through multiple mechanisms. Leukemia, 2018. https://doi.org/10.1038/s41375-018-0103-3 ↩

  2. The ENCODE Project Consortium. An integrated encyclopedia of DNA elements in the human genome. Nature, 2012. https://doi.org/10.1038/nature11247 ↩

  3. Gemma Satterthwaite, Sheila E. Francis, Kim Suvarna, et al. Differential gene expression in coronary arteries from patients presenting with ischemic heart disease: Further evidence for the inflammatory basis of atherosclerosis. American Heart Journal, 2005. https://doi.org/10.1016/j.ahj.2004.10.025 ↩

  4. Mouse Genome Sequencing Consortium. Initial sequencing and comparative analysis of the mouse genome. Nature, 2002. https://doi.org/10.1038/nature01262 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.