Working Your Own Molecular Data from the Command Line
By the end of this guide you will have a pinned software environment and verified copies of your own data files. You will also have indexes on everything that supports one, plus a set of commands that answer concrete questions about your genome and transcriptome in seconds. You need a Unix shell, which means macOS, Linux or WSL2 on Windows. You also need roughly 1 TB of free disk if you hold whole-genome sequencing data in both aligned and unaligned form, and 16 GB of RAM as a practical floor. Familiarity with pipes, redirection, and a text editor is assumed. Everything below is measurement and interpretation. Nothing here is a diagnosis, and questions about what a finding means for your health belong with a clinician or genetic counselor.
1. Build a pinned environment before you touch data
The single largest source of irreproducible results in personal genomics is an environment that drifted. Install tools with micromamba or mamba from the conda-forge and bioconda channels, and write the environment to a file you keep under version control. Pre-configured images exist for the same reason: they remove the assembly step and give you a known-good stack on day one 1.
# install micromamba, then:
cat > env.yml <<'EOF'
name: pmp
channels: [conda-forge, bioconda]
dependencies:
- samtools=1.20
- bcftools=1.20
- htslib=1.20
- bedtools=2.31.1
- seqkit=2.8.2
- csvtk=0.30.0
- parallel=20240522
- python=3.11
EOF
micromamba create -y -f env.yml
micromamba activate pmp
Pin exact versions. samtools and bcftools share htslib, and a mismatched pair produces confusing index errors. When you finish an analysis, record the state with micromamba env export --explicit > env.lock.txt and keep that file next to the results.
For anything you intend to rerun in a year, prefer a container. A Docker or Apptainer image with a digest pin is the only form of “same software” that survives a channel rebuild. Graphical wrappers exist and lower the entry barrier for specific pipelines 2. We use them for exploration and drop to the shell for anything we need to repeat.
2. Verify and inventory the files you received
Before analysis, confirm that the bytes on your disk are the bytes that left the lab. Checksums catch truncated downloads, which are common with files in the tens of gigabytes.
md5sum -c manifest.md5 # or sha256sum -c
Then identify each file rather than trusting its extension. htsfile reads magic bytes and tells you what you have.
htsfile * # e.g. "sample.cram: CRAM version 3.0 sequence data"
seqkit stats -a sample_R1.fastq.gz
samtools quickcheck -v *.bam *.cram && echo "EOF blocks intact"
samtools quickcheck verifies that the compressed stream terminates properly. It is fast and it catches the failure mode where a copy stopped halfway and every downstream tool reports a vague parse error.
Record what you have in a plain text inventory: file, size, checksum, and the read group or sample name inside it. Small helper collections such as bit cover many of these one-line summary tasks without writing your own scripts 3.
3. Know which reference genome your data is on
This matters more than any other single fact about your files. Coordinates are meaningless without a build. A variant at chr7:117,559,590 is one gene on GRCh38 and a different position on GRCh37.
Read the headers rather than guessing.
samtools view -H sample.cram | grep '^@SQ' | head
samtools view -H sample.cram | grep -E '^@PG|^@RG'
bcftools view -h sample.vcf.gz | grep -E '^##reference|^##contig' | head
Two things to check. First, the build: GRCh38 contigs include chrEBV and hundreds of alt and decoy contigs, while GRCh37 does not. Second, the naming style: chr1 versus 1. Mixing the two silently produces empty intersections, because bedtools and bcftools treat them as different chromosomes.
If you must convert, use a chromosome rename map instead of sed on the whole file.
printf "1\tchr1\n2\tchr2\n" > rename.txt # extend for all contigs
bcftools annotate --rename-chrs rename.txt -Oz -o renamed.vcf.gz in.vcf.gz
Lifting coordinates between builds is a separate operation with real failure rates. Use CrossMap or picard LiftoverVcf with the proper chain file, and expect a small fraction of positions to fail or flip strand. Never hand-edit coordinates.
4. Index everything, then query by region
Indexes turn a 40 GB file into a random-access database. Nothing in personal genomics is pleasant without them.
samtools faidx GRCh38.fa # .fai
samtools index -@ 4 sample.cram # .crai
bcftools index -t sample.vcf.gz # .tbi
bgzip -@ 4 regions.bed && tabix -p bed regions.bed.gz
One rule that catches everyone: tabix requires BGZF compression. Plain gzip will not do. A file made with gzip has the right extension and the wrong internal structure. Recompress it.
zcat plain.vcf.gz | bgzip -@ 4 -c > proper.vcf.gz && tabix -p vcf proper.vcf.gz
Now the queries are fast. To look at read-level evidence at one position, and to pull the variants in one gene:
samtools view -T GRCh38.fa sample.cram chr7:117559590-117559620 | head
samtools depth -a -r chr7:117559590-117559620 sample.cram
bcftools view -r chr17:43044295-43125483 -f PASS sample.vcf.gz | bcftools query \
-f '%CHROM\t%POS\t%REF\t%ALT\t%QUAL\t[%GT\t%DP]\n'
CRAM needs the reference to decode, so pass -T or set REF_PATH. That dependency is the price of the compression.
5. Normalize variants before you compare anything
Two files can describe the same variant with different text. Indels in particular have multiple valid left-alignments. Any comparison across your VCF and an external database needs normalization first, or you will get false mismatches.
bcftools norm -m -any -f GRCh38.fa -Oz -o norm.vcf.gz sample.vcf.gz
bcftools index -t norm.vcf.gz
-m -any splits multiallelic records into one line per alternate allele. -f left-aligns and trims against the reference. Do both, in that order, and check the count change:
bcftools stats sample.vcf.gz | grep -E '^SN'
bcftools stats norm.vcf.gz | grep -E '^SN'
A whole-genome callset from a single person typically reports on the order of four to five million variants, most of them common. If your numbers are wildly different, suspect a filtered subset or an exome rather than a genome.
Be careful with gVCF files. A gVCF contains non-variant blocks with <NON_REF> alleles, and treating it as a VCF will inflate your record counts and confuse annotation tools. Convert with your caller’s own tool before anything else.
Annotation is its own step, and the interpretation that follows belongs to a clinician. bcftools annotate can attach rsIDs and population frequencies from a bgzipped, tabix-indexed source:
bcftools annotate -a gnomad.sites.vcf.gz -c ID,INFO/AF -Oz -o annotated.vcf.gz norm.vcf.gz
6. Treat tabular data with tabular tools
RNA-seq counts and proteomics panels arrive as delimited text, and so do blood biomarkers and continuous glucose exports. The shell handles these well up to a few hundred megabytes. Format conversion and field-level cleaning are the everyday substance of this work, and doing them carefully is what keeps downstream statistics accurate 4.
csvtk is header-aware, which makes it safer than raw awk for named columns.
csvtk headers -t counts.tsv
csvtk filter2 -t -f '$tpm > 1' counts.tsv | csvtk sort -t -k tpm:nr | head -20
csvtk join -t -f gene_id counts.tsv gene_annotation.tsv > counts_annot.tsv
For CGM data, the export is usually a timestamped CSV with a device header block on top. Strip the header, parse the timestamp, and compute interval statistics.
tail -n +2 cgm.csv \
| csvtk cut -f timestamp,glucose_mgdl \
| csvtk mutate2 -n hour -e '${timestamp}[11:13]' \
| csvtk summary -g hour -f glucose_mgdl:mean,glucose_mgdl:stdev
Know when to stop. Once you need joins across three tables, missing-value logic, or any model fit, move to Python with polars or R with data.table. Shell pipelines are excellent at extraction and filtering and poor at statistics.
7. Make long jobs finish
Most personal-scale work is I/O bound. The CPU is usually not the limit. The first lever is threading inside the tools themselves.
samtools sort -@ 8 -m 2G -T /scratch/tmp -o sorted.bam in.bam
samtools view -@ 8 -T GRCh38.fa -C -o sample.cram sample.bam
Note -m is per thread. Eight threads at 2 GB will use roughly 16 GB, and the common out-of-memory failure comes from forgetting that. Set -T to a directory with real free space, because the default temporary location is often small.
The second lever is running independent jobs at once. GNU parallel over chromosomes is usually the right granularity.
seq 1 22 | parallel -j 6 --joblog run.log \
'bcftools view -r chr{} -f PASS norm.vcf.gz -Oz -o chunks/chr{}.vcf.gz'
On disk: converting BAM to CRAM typically cuts size by roughly half at the same information content, and more if you accept quality-score binning. We keep CRAM plus the reference and regenerate BAM on demand. Keep the original FASTQ or unaligned CRAM somewhere cold, because realignment against a future reference is the one operation you cannot redo without it.
8. Script it, log it, and write down what you did
Every command that produced a file you care about should live in a script. Start every script the same way.
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
set -e stops on error, set -u catches unset variables, and set -o pipefail makes a failing tool in the middle of a pipe fail the whole pipeline. Without pipefail a broken samtools view feeding a grep will return success and you will analyze an empty file.
For anything with more than about five steps, use a workflow manager. Snakemake or Nextflow gives you resumption after a failure and a dependency graph you can inspect. A Makefile is a reasonable minimum if you want no new dependencies.
Keep a README.md per project with the reference build, the tool versions, the source of each input, and the date. Publishing well-documented command-line tools alongside interactive front ends is the norm in the field for exactly this reason: the command line is what gets rerun and audited 5.
Common problems
Chromosome naming mismatch. bedtools intersect returns zero lines and no error. Compare cut -f1 file.bed | sort -u against samtools view -H x.cram | grep -c 'SN:chr'. Fix with a rename map, not sed.
Wrong reference build. Symptoms include annotation hits in the wrong gene and an unusual fraction of variants absent from population databases. Check the @SQ lines and the presence of chrEBV or hs38d1 decoy contigs.
tabix refuses to index. The file is gzip rather than BGZF. Recompress with bgzip. Verify with htsfile file.vcf.gz, which should report BGZF.
Sort order errors. bcftools requires coordinate-sorted, and sort on text files depends on locale. Set LC_ALL=C before any sort on genomic text, otherwise you get a different ordering on different machines.
Out of temporary space. samtools sort and sort both spill to disk. Set -T and TMPDIR explicitly to a large volume and watch with df -h.
Silent truncation. A copy interrupted at 90 percent gives you a file that most tools read partway and then abandon. samtools quickcheck and checksum verification catch this before you waste a day.
Multiallelic and normalization mismatches. Your variant is present in both files and appears in neither intersection. Run bcftools norm -m -any -f ref.fa on both sides first.
Reading meaning into a single call. A variant call with depth of 4 reads and a low quality score is a hypothesis. It is not a fact. Look at the reads with samtools view or IGV before believing anything. Any variant that appears to carry clinical weight needs confirmatory testing in an accredited laboratory and a conversation with a clinician or genetic counselor. Research-grade sequencing is not a diagnostic result.
Assuming an expression value is stable. A single RNA-seq timepoint reflects the tissue, the time of day, and what you did the day before. Comparisons are most useful against your own prior measurements under similar conditions.
Footnotes
-
Konstantinos Krampis, Tim Booth, Brad Chapman, et al. Cloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community. BMC Bioinformatics, 2012. https://doi.org/10.1186/1471-2105-13-42 ↩
-
Martti Vasar, John Davison, Lena Neuenkamp, et al. User‐friendly bioinformatics pipeline gDAT (graphical downstream analysis tool) for analysing rDNA sequences. Molecular Ecology Resources, 2021. https://doi.org/10.1111/1755-0998.13340 ↩
-
Michael Lee. bit: a multipurpose collection of bioinformatics tools. F1000Research, 2022. https://doi.org/10.12688/f1000research.79530.1 ↩
-
J. B. Brown. Fundamental Bioinformatic and Chemoinformatic Data Processing. Methods in Molecular Biology, 2018. https://doi.org/10.1007/978-1-4939-8639-2_3 ↩
-
Nicholas R. Waters, Florence Abram, Fiona Brennan, et al. Easy phylotyping of Escherichia coli via the EzClermont web app and command-line tool. Access Microbiology, 2020. https://doi.org/10.1099/acmi.0.000143 ↩
Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.