Running DESeq2 on Your Own RNA-Seq Data
By the end of this guide you will have a gene-level count matrix and a fitted negative binomial model. You will also have a table of log fold changes with shrunken effect sizes and adjusted p-values, and a set of diagnostic plots that tell you whether the table means anything. You need FASTQ files from at least two groups of samples, a reference transcriptome and matching annotation, R 4.3 or newer with DESeq2, tximport, and apeglm installed, and a colData table describing each sample. Plan on roughly 2 GB of RAM per concurrent salmon job and a few minutes of model fitting for a typical 20,000-gene by 10-sample matrix. If you only have one sample, stop here: differential expression requires replication, and no amount of statistics will recover it.
1. Quantify to raw counts without converting to TPM
DESeq2 models counts with a negative binomial distribution, and the variance of that distribution depends on the count scale. TPM and FPKM have already divided out library size and gene length, which destroys the information the dispersion estimator needs. So the answer to “does DESeq2 require raw counts” is yes, with one accepted exception: estimated counts from a transcript quantifier passed through tximport, which also supplies an average-transcript-length offset per gene and sample.
We use salmon for quantification. It is fast, it corrects for fragment GC bias and positional bias, and its transcript-level output aggregates cleanly to genes.
salmon index -t gencode.v45.transcripts.fa.gz \
-i salmon_idx_v45 --gencode -k 31 -p 8
for s in S01 S02 S03 S04 S05 S06; do
salmon quant -i salmon_idx_v45 -l A \
-1 fastq/${s}_R1.fastq.gz -2 fastq/${s}_R2.fastq.gz \
--validateMappings --gcBias --seqBias \
--numBootstraps 30 -p 8 -o quant/${s}
done
Use -k 31 for reads of 75 bp or longer and drop to -k 23 or so for short reads. The --gcBias flag matters more than people expect when libraries were prepared on different days. Check the mapping rate in quant/${s}/logs/salmon_quant.log before going further. Rates below about 70% for a human polyA library usually mean rRNA or genomic DNA contamination, or an index built from the wrong annotation release.
Then import into R, collapsing transcripts to genes.
library(tximport); library(DESeq2)
tx2gene <- read.csv("tx2gene_v45.csv") # columns: TXNAME, GENEID
files <- file.path("quant", coldata$sample, "quant.sf")
names(files) <- coldata$sample
txi <- tximport(files, type = "salmon", tx2gene = tx2gene,
countsFromAbundance = "no", ignoreTxVersion = FALSE)
Keep countsFromAbundance = "no" so that DESeqDataSetFromTximport can use the length offsets directly. Set ignoreTxVersion = TRUE only if your tx2gene table lacks version suffixes. A silent version mismatch produces a matrix of mostly zeros, which is easy to catch by checking sum(txi$counts) against total mapped reads.
An alignment-based route with STAR plus featureCounts is equally valid and gives you a BAM file for visual inspection. It is slower and it ignores transcript-length differences between samples. We use salmon by default and keep STAR for cases where we need to look at coverage over a specific locus.
2. Build the DESeqDataSet and choose a design formula
The design formula is the single most consequential decision in the analysis. It declares which sources of variation the model will estimate and which it will leave in the residual.
coldata$condition <- factor(coldata$condition, levels = c("baseline", "fasted"))
coldata$batch <- factor(coldata$batch)
dds <- DESeqDataSetFromTximport(txi, colData = coldata,
design = ~ batch + condition)
Two rules. First, make the variable of interest last in the formula, because results() defaults to the last coefficient. Second, set the reference level explicitly with factor(levels = ...) or relevel(). R orders levels alphabetically, so “baseline” versus “fasted” happens to work and “control” versus “acute” does not.
For repeated measures on one person, or on a small cohort sampled at several timepoints, use a paired design: ~ subject + timepoint. This absorbs each individual’s baseline expression into a subject term and tests the within-subject change. It is the right model for longitudinal personal data and it is far more sensitive than a between-group comparison on the same sample count. The cost is that you can no longer test anything that is constant within a subject, such as sex or genotype. Such a variable is collinear with the subject term and the design matrix will not be full rank.
3. Filter low counts, then look at the data before you test anything
Genes with almost no reads contribute noise and inflate the multiple testing burden without contributing power. Filter on a rule tied to your smallest group size. Do not filter on an arbitrary row sum.
smallest_group <- min(table(coldata$condition))
keep <- rowSums(counts(dds) >= 10) >= smallest_group
dds <- dds[keep, ]
nrow(dds)
For a human whole-blood or tissue library this typically leaves 13,000 to 16,000 genes. Prespecifying the filter is good practice and it is a formal requirement in regulated settings. The R-ODAF framework for regulatory omics applications writes down both the low-count filter and the effect-size criteria in advance, so the gene list is not a function of choices made after seeing the results 1.
Now transform and plot. The variance-stabilizing transformation gives you something close to homoskedastic on the log scale, which is what PCA and clustering assume.
vsd <- vst(dds, blind = TRUE)
plotPCA(vsd, intgroup = c("condition", "batch"))
library(pheatmap)
sampleDist <- dist(t(assay(vsd)))
pheatmap(as.matrix(sampleDist))
Read the PCA. If PC1 separates by batch or by RNA integrity rather than by condition, the differential expression table will be dominated by that. If one sample sits far from everything else, find out why before you decide what to do with it. blind = TRUE is correct here because you want a transformation that has not been told the design. For a version that borrows dispersion information from the design, use blind = FALSE, and use it only for visualization.
4. Fit the model
One call runs three stages.
dds <- DESeq(dds)
The first stage estimates size factors by the median-of-ratios method. For each gene, DESeq2 takes the ratio of each sample’s count to the gene’s geometric mean across samples, then takes the median of those ratios within a sample. The median makes the estimate insensitive to a handful of very highly expressed genes, which is why a single dominant globin or rRNA gene shifts a library’s size factor much less than a naive total-count normalization would.
The second stage estimates dispersion, which is the extra variance beyond Poisson shot noise. DESeq2 first fits a gene-wise maximum likelihood dispersion, then fits a smooth trend of dispersion against mean expression, then shrinks each gene-wise estimate toward that trend by empirical Bayes. Shrinkage is what makes analyses with three or four replicates per group usable at all. Always plot the result.
plotDispEsts(dds)
You want a decreasing cloud of black gene-wise estimates, a red trend line through it, and blue shrunken values pulled toward the line. Genes flagged as dispersion outliers stay at their own high estimate and are therefore treated conservatively.
The third stage fits a negative binomial generalized linear model with a log link and tests each coefficient with a Wald test. That is the answer to “what statistical test does DESeq2 use”: a Wald test on the GLM coefficient, with Benjamini-Hochberg correction across genes. For designs with more than two levels, or when you want to test several coefficients at once, use a likelihood ratio test instead.
dds_lrt <- DESeq(dds, test = "LRT", reduced = ~ subject)
5. Extract results and shrink the fold changes
res <- results(dds, contrast = c("condition", "fasted", "baseline"),
alpha = 0.05)
summary(res)
The contrast argument makes the comparison explicit and readable months later. Keep alpha equal to the false discovery rate you will report, because results() uses it to tune independent filtering. That filtering step removes low-mean genes from the multiple testing correction to increase power, and it is why some genes have a p-value but NA in padj.
Raw log2 fold changes for low-count genes are wildly unstable. Shrink them.
resultsNames(dds)
res_shrunk <- lfcShrink(dds, coef = "condition_fasted_vs_baseline",
type = "apeglm")
Use apeglm as the default. It shrinks weakly informed fold changes toward zero while leaving well-estimated large effects nearly untouched, which makes the ranked list much more useful than ranking on unshrunken values. The p-values do not change. Rank by shrunken effect size, filter on padj, and never rank by p-value alone if you care about magnitude.
If you want a test against a minimum biological effect rather than against zero, ask for it directly.
res_thr <- results(dds, contrast = c("condition", "fasted", "baseline"),
lfcThreshold = log2(1.5), altHypothesis = "greaterAbs")
This is more defensible than post hoc screening on a fold-change cutoff, because the null hypothesis includes the threshold and the p-values account for it.
6. Write out the table and sanity-check the top hits
res_df <- as.data.frame(res_shrunk)
res_df$gene <- rownames(res_df)
res_df <- res_df[order(res_df$padj), ]
write.csv(res_df, "deseq2_fasted_vs_baseline.csv", row.names = FALSE)
plotCounts(dds, gene = res_df$gene[1], intgroup = "condition")
Plot normalized counts for your top five genes. A real effect looks like consistent separation across replicates. An artifact usually looks like one sample carrying the entire signal. This takes two minutes and it catches more errors than any summary statistic.
7. If you would rather work in Python
PyDESeq2 reimplements the core DESeq2 method in Python and reproduces the size factor, dispersion, and Wald test results closely enough for practical use 2. The API is close to scikit-learn in shape.
from pydeseq2.dds import DeseqDataSet
from pydeseq2.ds import DeseqStats
dds = DeseqDataSet(counts=counts_df, metadata=meta_df,
design="~batch + condition", refit_cooks=True)
dds.deseq2()
stat = DeseqStats(dds, contrast=["condition", "fasted", "baseline"])
stat.summary()
stat.lfc_shrink(coeff="condition[T.fasted]")
counts_df must be samples by genes, integer, with the sample index matching meta_df. This is transposed relative to the R convention and it is the most common source of confusion. We use PyDESeq2 when the analysis already lives in a Python pipeline and the R implementation when we want the full set of Bioconductor diagnostics and downstream annotation packages.
8. On edgeR, limma, and whether the choice matters
Direct comparisons generally find substantial overlap between DESeq2, edgeR, and limma-voom on the same data. The differences concentrate in genes near the significance boundary 3. edgeR tends to call more genes at a fixed nominal FDR, which reads as higher sensitivity and as looser control depending on the dataset 4. Studies in plant systems report broad agreement in the ranked gene lists alongside those calling-threshold differences 5.
Our view: pick one method, prespecify it, and report it. Running all three and reporting the union is a way of inflating the false discovery rate without saying so. If you want to know how a given pipeline behaves at your sample size and correlation structure, simulate data that carries realistic inter-gene correlation rather than trusting independent-gene simulations 6.
Common problems
“Model matrix is not full rank.” Two variables in your design carry the same information. The cause is almost always a paired design with a subject-constant covariate, or a batch that lines up perfectly with condition. Drop the redundant term, or if the confounding is complete, accept that the effect is not identifiable from these samples.
Every padj is NA. Either independent filtering removed the gene for low mean expression, or the gene was flagged as a count outlier by Cook’s distance. With seven or more replicates per group DESeq2 replaces outlier counts with the trimmed mean instead of discarding the gene. With fewer, it sets the p-value to NA. Inspect with mcols(res)$maxCooks.
Dispersion fit fails or looks nothing like the expected trend. Try DESeq(dds, fitType = "local"). Only if that also fails, use fitType = "mean". A pathological dispersion plot often means the samples are not exchangeable within groups, so check the PCA again.
Many zeros and unstable size factors. Use estimateSizeFactors(dds, type = "poscounts"), which computes the geometric mean over positive counts only. This matters for low-input or single-cell-like data. It does not matter for standard bulk libraries.
Batch correction applied to the counts. Do not run limma::removeBatchEffect and feed the output to DESeq2. Put batch in the design formula instead. Use the corrected matrix only for PCA and heatmaps, and say so in the figure legend.
Whole blood composition confounding everything. In blood RNA, a shift in neutrophil or reticulocyte fraction changes thousands of genes at once, and it will dominate any condition effect. Globin depletion during library prep helps with the reticulocyte part. Cell-type deconvolution of the mixture and adding the estimated proportions as covariates helps with the rest.
Interpreting a gene list as a finding about your health. A differential expression table describes which transcripts changed between the conditions you sampled. It does not establish cause, and it does not diagnose anything. Multi-omics work that reaches clinical stratification does so by combining expression with genomic, proteomic, and clinical layers in cohorts. It does not come from one expression contrast 7. Anything that touches a medical decision belongs with a clinician who can see your full record.
Footnotes
-
M.C. Verheijen, T.W. Gant, W. Tong, et al. R-ODAF: Omics data analysis framework for regulatory application. Toxicology Letters, 2021. https://doi.org/10.1016/s0378-4274(21)00539-7 ↩
-
Boris Muzellec, Maria Teleńczuk, Vincent Cabeli, et al. PyDESeq2: a python package for bulk RNA-seq differential expression analysis. Bioinformatics, 2023. https://doi.org/10.1093/bioinformatics/btad547 ↩
-
Shiyi Liu, Zitao Wang, Ronghui Zhu, et al. Three Differential Expression Analysis Methods for RNA Sequencing: limma, EdgeR, DESeq2. Journal of Visualized Experiments, 2021. https://doi.org/10.3791/62528 ↩
-
Mostafa Rezapour. Tool choice matters: Evaluating edgeR vs. DESeq2 for sensitivity, robustness, and cross-study performance. PLOS One, 2026. https://doi.org/10.1371/journal.pone.0353788 ↩
-
Grażyna Niedziela, Alicja Szabelska-Beręsewicz, Joanna Zyprych-Walczak, et al. Application of edgeR and DESeq2 methods in plant experiments based on RNA-seq technology. Biometrical Letters, 2022. https://doi.org/10.2478/bile-2022-0009 ↩
-
Jianing Yang, Gregory R. Grant, Thomas G. Brooks. Generating correlated data for omics simulation. PLOS Computational Biology, 2025. https://doi.org/10.1371/journal.pcbi.1013392 ↩
-
Chi-Sheng Wu, Hsin-Pai Li, Chia-Hsun Hsieh, et al. Integrated multi-omics analyses of oral squamous cell carcinoma reveal precision patient stratification and personalized treatment strategies. Cancer Letters, 2025. https://doi.org/10.1016/j.canlet.2025.217482 ↩
Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.