Skip to content

What GATK's ReadPosRankSum measures and how to filter on it

Oak
Underwater reef scene with drifting translucent ribbons, some glowing beads centered in them, others clustered at the ribbons' frayed ends.

ReadPosRankSum is a variant-level annotation in a GATK VCF that asks one question: do the reads carrying the alternate allele place that allele closer to the ends of the reads than the reference-carrying reads place the reference allele? It is the z-approximation of a Mann-Whitney U rank sum test over per-read positions, so zero means no difference and negative values mean the alternate allele clusters near read ends. GATK’s recommended germline hard filters discard SNPs below -8.0 and indels below -20.0. The single most important practical fact is that the annotation only exists at sites with both reference and alternate reads, so it is absent at homozygous-variant calls and your filter expression will silently skip them unless you handle that case.

What the annotation measures

HaplotypeCaller walks every read that informatively supports an allele at the site and records the offset of the variant within that read, measured as the distance to the nearer end of the read. Hard-clipped bases are excluded from that distance, and reads that fail base quality or mapping quality thresholds do not contribute. The result is two samples of distances: one from reference-supporting reads, one from alternate-supporting reads.

Those two samples go into a rank sum test rather than a comparison of means. Ranks are the right choice because read-position distributions are bounded, discrete and skewed, so a t-test on raw offsets would be badly behaved. What lands in the INFO field is the u-based z approximation, which is roughly interpretable as a standard normal deviate. A value of -8.0 is a very strong asymmetry. It is not a marginal one.

For indels the offset is computed for the event rather than a single base, and the tolerance in the recommended filter is much looser. That looseness is deliberate. Indel representation near read ends is genuinely noisy even for true variants.

Why alleles near read ends are suspect

Read ends are where alignment quality degrades. Base quality drops along the read, adapter and barcode remnants survive trimming, and the aligner has less sequence context to anchor a gapped alignment. A short indel in a homopolymer or short tandem repeat that falls in the last fifteen bases of a read is frequently placed there because the aligner had no better option.

Mapping artifacts produce the same signature. If reads from a paralogous locus or a segmental duplication pile up at your site, the divergent bases they carry tend to sit at a consistent offset, and that offset is often near an end because the rest of the read matched. A strongly negative ReadPosRankSum therefore flags sites where the alternate allele is supported only by reads that had a reason to be wrong there.

The annotation is one member of a panel that includes QD (quality normalized by depth), FS and SOR (strand bias), MQ and MQRankSum (mapping quality and its rank sum test). On its own, the annotation is not sufficient. Empirical work tuning these thresholds on targeted panels shows that the optimal cutoffs shift with capture design and depth. The defaults are a starting point rather than a law.1 Larger evaluations against ultra-deep sequencing confirm that combinations of these annotations separate true from false calls, rather than any single one.2

Where the value is undefined, and why that breaks filters

ReadPosRankSum compares two groups. At a homozygous-variant site there are no reference-supporting reads, so there is nothing to compare and HaplotypeCaller emits no ReadPosRankSum key at all. In a 30x human genome you will typically see four to five million short variants, and a substantial minority of them are homozygous for the alternate allele. That is a large fraction of your callset with the field missing.

VariantFiltration evaluates JEXL expressions. When an expression references a key that is absent, GATK logs a warning about an undefined variable and treats the variant as not matching the filter, so the call passes through unfiltered. That is the behavior you want here. The alternative is worse:

--missing-values-evaluate-as-failing

Setting that flag makes every homozygous-variant call fail your ReadPosRankSum filter. We do not use it with rank sum annotations for exactly that reason. If you want the flag for some other expression, put that expression in a separate VariantFiltration invocation.

The clean pattern is one filter name per condition, applied in a single pass:

gatk VariantFiltration \
  -R GRCh38.fa \
  -V snps.vcf.gz \
  -O snps.filtered.vcf.gz \
  --filter-name "QD2"    --filter-expression "QD < 2.0" \
  --filter-name "QUAL30" --filter-expression "QUAL < 30.0" \
  --filter-name "SOR3"   --filter-expression "SOR > 3.0" \
  --filter-name "FS60"   --filter-expression "FS > 60.0" \
  --filter-name "MQ40"   --filter-expression "MQ < 40.0" \
  --filter-name "MQRS-12.5" --filter-expression "MQRankSum < -12.5" \
  --filter-name "RPRS-8" --filter-expression "ReadPosRankSum < -8.0"

Indels and mixed sites take the looser set. Use QD < 2.0, QUAL < 30.0, FS > 200.0 and ReadPosRankSum < -20.0. Split the callset first with SelectVariants and -select-type SNP or -select-type INDEL, filter each, then merge with MergeVcfs. These are the thresholds used in published population-scale single-sample WGS work, and they are a reasonable default for one person’s genome.3

Looking at the distribution in your own data

Never accept a threshold you have not seen plotted against your own data. Export the annotations to a table:

gatk VariantsToTable \
  -V raw.vcf.gz \
  -F CHROM -F POS -F TYPE -F QD -F DP -F MQ \
  -F ReadPosRankSum -F MQRankSum -F FS -F SOR \
  --show-filtered -O annotations.tsv

In a healthy 30x germline callset the SNP histogram of ReadPosRankSum is centered near zero and roughly symmetric, with a thin left tail. If the whole distribution is shifted negative, suspect trimming or library preparation rather than individual variants. If your indel tail is heavy below -20, check whether your BAM was produced with a local realignment step and what the read length is.

Two useful cross-tabulations follow. First, ReadPosRankSum against DP: the negative tail should concentrate at low and at abnormally high depth. Second, the fraction of filtered calls that overlap a truth set such as GIAB HG002 or your own replicate sequencing. Replicate whole-genome runs on the same donor are the most direct way to see which of your calls are reproducible.4

If the annotation is missing across the board, and not merely at homozygous sites, you are probably looking at a VCF that lost it in processing. Rank sum annotations survive gVCF workflows only if HaplotypeCaller wrote them and joint genotyping carried them forward. GATK summarizes rank sum values across samples during joint genotyping, which dilutes any single-sample signal, so a cohort VCF’s ReadPosRankSum is a weaker filter than a single-sample one. You can recompute it from the alignment with VariantAnnotator and -A ReadPosRankSumTest, supplying the original BAM or CRAM.

When to stop hard-filtering

Hard filters are a threshold on a handful of hand-picked annotations, and they throw away the correlations between those annotations. For a single deeply sequenced genome they remain the practical choice, since VQSR needs many samples or a well-matched resource set to behave. Best-practice pipelines for germline short variants still specify this set of thresholds for the single-sample case.5

Machine-learned filters do better when you have the training data, because they use ReadPosRankSum jointly with depth, strand bias and quality rather than in isolation.6 Somatic work has moved further still. Mutect2 and FilterMutectCalls use ReadPosition, the median distance from the variant to the nearest read end, evaluated inside a probabilistic model rather than as a fixed cutoff. Tumor-only calling in particular depends on aggressive position and orientation filters to suppress artifacts.7

Whatever you use, treat a ReadPosRankSum failure as a statement about the sequencing. The failure does not describe your biology. Filtered calls are calls you cannot trust from this data. Any variant you intend to act on belongs in front of a clinical geneticist with orthogonal confirmation. A spreadsheet of your own filtered VCF is not the place for it.

Questions people also ask

What does a negative ReadPosRankSum mean? The alternate allele sits closer to the ends of its supporting reads than the reference allele sits in its own. Large negative values suggest the alternate reads are artifacts of end-of-read alignment error.

Why is ReadPosRankSum undefined for some variants? The test needs both reference-supporting and alternate-supporting reads. Homozygous-variant sites have no reference reads, so GATK omits the key entirely and your JEXL expression cannot evaluate.

Should I use --missing-values-evaluate-as-failing? Not with ReadPosRankSum or MQRankSum. It would fail every homozygous-variant call in your genome. Leave the default, which passes variants whose annotation is absent.

What thresholds should I use? Start at -8.0 for SNPs and -20.0 for indels, then plot your own distribution. Panel and exome data with unusual depth profiles often need different cutoffs.18

Is ReadPosRankSum the same as ReadPosition? No. ReadPosition is the median distance from the variant to the nearest read end, used by Mutect2’s somatic filters. ReadPosRankSum is a two-sample rank sum comparison and is a germline annotation.

Footnotes

  1. Simona De Summa, Giovanni Malerba, Rosamaria Pinto, et al. GATK hard filtering: tunable parameters to improve variant calling for next generation sequencing targeted gene panel data. BMC Bioinformatics, 2017. https://doi.org/10.1186/s12859-017-1537-8 ↩ ↩2

  2. Toshihiro Kishikawa, Yukihide Momozawa, Takeshi Ozeki, et al. Empirical evaluation of variant calling accuracy using ultra-deep whole-genome sequencing data. Scientific Reports, 2019. https://doi.org/10.1038/s41598-018-38346-0 ↩

  3. Lucy Crooks, Johnathan Cooper-Knock, Paul R. Heath, et al. Identification of single nucleotide variants in the Moroccan population by whole-genome sequencing. BMC Genetics, 2020. https://doi.org/10.1186/s12863-020-00917-4 ↩

  4. Marcus Høy Hansen, Charlotte Guldborg Nyvold. Replicate whole-genome next-generation sequencing data derived from Caucasian donor saliva samples. Data in Brief, 2021. https://doi.org/10.1016/j.dib.2021.107349 ↩

  5. V. P. Snijesh, K Shivakumar, K. M. Punya, et al. Best Practices for Variant Calling Using the Genome Analysis Toolkit. Next-Generation Sequencing, 2025. https://doi.org/10.1201/9781003354062-4 ↩

  6. Vladimir Kovacevic, Predrag Obradovic. Smart variant filtering. International Journal of Data Mining and Bioinformatics, 2021. https://doi.org/10.1504/ijdmb.2021.126836 ↩

  7. Yu-Chin Hsu, Yu-Ting Hsiao, Tzu-Yuan Kao, et al. Detection of Somatic Mutations in Exome Sequencing of Tumor-only Samples. Scientific Reports, 2017. https://doi.org/10.1038/s41598-017-14896-7 ↩

  8. Namshin Kim, Kyoung Hyoun Kim, Won-Jun Lim, et al. Whole Exome Sequencing Identifies Novel De Novo Variants Interacting with Six Gene Networks in Autism Spectrum Disorder. Genes, 2020. https://doi.org/10.3390/genes12010001 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.