Skip to content

How to Download and Parse the Human Metabolome Database (HMDB)

Oak
A river of glowing particles splits through a coral-like filter into many colored streams running into terraced pools at night.

By the end of this guide you will have a local, versioned copy of the Human Metabolome Database as a set of Parquet files and a DuckDB database. It holds one row per metabolite plus exploded child tables for concentrations, pathways and cross-references. It also includes an index that lets you join HMDB annotations to your own metabolite measurements by InChIKey. You need roughly 40 GB of free disk and DuckDB. You also need Python 3.10 or later with lxml and pyarrow, on a machine with at least 8 GB of RAM. The full metabolite XML is a single multi-gigabyte document, so the entire approach here is streaming: nothing in this guide loads the file into memory. Expect the parse to take 10 to 30 minutes on a laptop.

1. Confirm you want this HMDB, and read the license

Three different resources share the acronym, and search engines mix them freely. The Human Metabolome Database at hmdb.ca is a knowledgebase of small-molecule metabolites found in the human body, with chemical, clinical and molecular biology data for each entry 1. HmtDB is a wholly separate project on human mitochondrial genome variability, built for population genetics and variant interpretation 2. It is sometimes abbreviated the same way, and it is what you want if your question is about mtDNA haplogroups rather than metabolites 3. A third HMDB is a curated database of genes involved in hydrocarbon monooxygenation, aimed at environmental microbiology 4. If you downloaded a file and the identifiers look like gene names rather than HMDB accessions, you have the wrong one.

The metabolome database is free to download, and free for academic use. Commercial use requires a license from the maintainers, and that includes internal use at a company even when you never redistribute the data. Check the current terms on the download page before you build anything on top of it. This matters more than people expect, because HMDB’s curated content is expensive to reproduce and the license is enforced.

2. Download the files you need and no others

The downloads page offers the whole database plus several subsets. The decision worth making up front is which of these you will use, because the spectral archives are far larger than the metabolite records and most readers do not need them.

For annotation work, take hmdb_metabolites.zip. This is the complete set of metabolite records in XML, on the order of 220,000 entries, and it expands to several gigabytes. The biospecimen subsets have the same schema and contain only metabolites reported in that fluid. The subsets are the following files.

  • serum_metabolites.zip
  • urine_metabolites.zip
  • csf_metabolites.zip
  • saliva_metabolites.zip
  • feces_metabolites.zip
  • sweat_metabolites.zip

If you are annotating a plasma panel, the serum subset is a tenth of the size and is the file we start with when prototyping a parser. Move to the full file once the code works.

structures.zip gives you an SDF with the molecular structures, which is what you want for RDKit work or for regenerating InChIKeys yourself. hmdb_proteins.zip covers the enzymes and transporters linked to metabolites. The MS/MS archives come in two flavors, experimental and predicted, and the predicted set is much larger. Download those only if you are doing untargeted spectral matching, and read step 7 first.

mkdir -p hmdb/2026-09 && cd hmdb/2026-09

curl -fL --retry 5 --retry-delay 10 -C - \
  -o hmdb_metabolites.zip \
  https://hmdb.ca/system/downloads/current/hmdb_metabolites.zip

curl -fL -o serum_metabolites.zip \
  https://hmdb.ca/system/downloads/current/serum_metabolites.zip

shasum -a 256 *.zip | tee SHA256SUMS
unzip -o hmdb_metabolites.zip
ls -la hmdb_metabolites.xml

Two notes on the download itself. Use -C - so an interrupted transfer resumes instead of restarting, because these files are slow to pull and the server will occasionally close the connection. Record your own checksums in a file next to the data, since HMDB publishes releases under a current path that changes in place. A checksum plus a dated directory is the only way you will later know which release produced a given annotation.

3. Inspect the schema before you write a parser

Never open a 5 GB XML file in an editor. Look at the first record instead. It tells you the namespace, the element name for a record, and the field names.

head -c 20000 hmdb_metabolites.xml | xmllint --format - 2>/dev/null | head -120

You will see a root <hmdb xmlns="http://www.hmdb.ca"> containing repeated <metabolite> elements. Each metabolite carries scalar fields such as the following.

  • accession
  • name
  • chemical_formula
  • average_molecular_weight
  • monisotopic_molecular_weight
  • smiles
  • inchi
  • inchikey
  • cas_registry_number

It also carries the following nested blocks.

  • secondary_accessions
  • taxonomy with a chemical classification
  • biological_properties containing cellular_locations, biospecimen_locations, tissue_locations and pathways
  • normal_concentrations
  • abnormal_concentrations
  • diseases
  • protein_associations
  • a block of external database identifiers (kegg_id, pubchem_compound_id, chebi_id, chemspider_id, drugbank_id and foodb_id)

This record layout is the XML form of what HMDB calls a MetaboCard, the per-metabolite card visible in the web interface 5.

The namespace is the first thing that breaks naive parsers. Every element tag arrives as {http://www.hmdb.ca}metabolite, so a match on "metabolite" finds nothing and your script silently produces zero rows. Count records early so you notice:

grep -c "<metabolite>" hmdb_metabolites.xml

4. Stream the XML into Parquet

The parse strategy is lxml.etree.iterparse on end events for metabolite elements, flattening each record, then clearing the element and its consumed ancestors so memory stays flat. Without the clearing step your process grows to the size of the file and dies.

# hmdb_to_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq
from lxml import etree

NS = "{http://www.hmdb.ca}"
SCALARS = [
    "accession", "name", "chemical_formula", "average_molecular_weight",
    "monisotopic_molecular_weight", "iupac_name", "traditional_iupac",
    "cas_registry_number", "smiles", "inchi", "inchikey", "state",
    "kegg_id", "pubchem_compound_id", "chebi_id", "chemspider_id",
    "drugbank_id", "foodb_id", "status",
]

def text(el, tag):
    child = el.find(NS + tag)
    if child is None or child.text is None:
        return None
    value = child.text.strip()
    return value or None

def texts(el, container, item):
    parent = el.find(NS + container)
    if parent is None:
        return []
    return [c.text.strip() for c in parent.findall(NS + item) if c.text]

def flatten(el):
    row = {k: text(el, k) for k in SCALARS}
    tax = el.find(NS + "taxonomy")
    if tax is not None:
        for k in ("super_class", "class", "sub_class", "direct_parent"):
            row["taxonomy_" + k] = text(tax, k)
    bio = el.find(NS + "biological_properties")
    if bio is not None:
        row["biospecimens"] = texts(bio, "biospecimen_locations", "biospecimen")
        row["cellular_locations"] = texts(bio, "cellular_locations", "cellular")
        row["tissues"] = texts(bio, "tissue_locations", "tissue")
    row["secondary_accessions"] = texts(el, "secondary_accessions", "accession")
    return row

def concentrations(el, kind):
    out = []
    parent = el.find(NS + kind)
    if parent is None:
        return out
    for c in parent.findall(NS + "concentration"):
        out.append({
            "accession": text(el, "accession"),
            "kind": kind,
            "biospecimen": text(c, "biospecimen"),
            "value": text(c, "concentration_value"),
            "units": text(c, "concentration_units"),
            "age": text(c, "subject_age"),
            "sex": text(c, "subject_sex"),
            "condition": text(c, "subject_condition"),
            "comment": text(c, "comment"),
        })
    return out

def run(xml_path, out_prefix, batch_size=5000):
    mets, concs = [], []
    met_writer = conc_writer = None
    ctx = etree.iterparse(xml_path, events=("end",), tag=NS + "metabolite")
    n = 0
    for _, el in ctx:
        mets.append(flatten(el))
        concs += concentrations(el, "normal_concentrations")
        concs += concentrations(el, "abnormal_concentrations")
        n += 1
        if len(mets) >= batch_size:
            met_writer, mets = _flush(mets, met_writer, f"{out_prefix}_metabolites.parquet")
            conc_writer, concs = _flush(concs, conc_writer, f"{out_prefix}_concentrations.parquet")
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]
    _flush(mets, met_writer, f"{out_prefix}_metabolites.parquet")
    _flush(concs, conc_writer, f"{out_prefix}_concentrations.parquet")
    print(f"parsed {n} metabolites")

def _flush(rows, writer, path):
    if not rows:
        return writer, []
    table = pa.Table.from_pylist(rows)
    if writer is None:
        writer = pq.ParquetWriter(path, table.schema)
    writer.write_table(table)
    return writer, []

if __name__ == "__main__":
    run("hmdb_metabolites.xml", "hmdb")

Two design choices are worth stating. We write in batches to a single ParquetWriter rather than accumulating a table, which keeps resident memory near constant. We keep list-valued fields as Arrow lists instead of joining them with a delimiter, because delimiters appear inside HMDB tissue and disease names and will corrupt any later split.

The el.getprevious() loop is the part people omit. Clearing the element releases its children, but the parent root still holds a reference to every sibling parsed so far. Deleting preceding siblings is what bounds memory.

5. Load into DuckDB and build the identifier index

Parquet is enough for scans, but you want joins and a stable index. DuckDB reads the Parquet directly and gives you SQL over it with no import step.

-- hmdb.sql
CREATE OR REPLACE TABLE metabolites AS
SELECT * FROM read_parquet('hmdb_metabolites.parquet');

CREATE OR REPLACE TABLE concentrations AS
SELECT * FROM read_parquet('hmdb_concentrations.parquet');

-- one row per (accession, alias) for lookup by any historic ID
CREATE OR REPLACE TABLE accession_alias AS
SELECT accession, accession AS alias FROM metabolites
UNION ALL
SELECT accession, UNNEST(secondary_accessions) AS alias FROM metabolites;

-- 14-character InChIKey skeleton, for matching across salts and stereo forms
CREATE OR REPLACE TABLE inchikey_index AS
SELECT accession, name, inchikey,
       SPLIT_PART(inchikey, '-', 1) AS inchikey_block1
FROM metabolites
WHERE inchikey IS NOT NULL;
duckdb hmdb.duckdb < hmdb.sql
duckdb hmdb.duckdb -c "SELECT COUNT(*) FROM metabolites;"

The alias table matters because HMDB renumbered accessions when it expanded from five-digit to seven-digit IDs. An older paper or vendor report may cite HMDB00122 while the current record is HMDB0000122. Zero-pad on lookup and check the alias table before concluding an ID is missing.

The first InChIKey block is the connectivity hash. Two entries sharing it are the same skeleton with different stereochemistry, protonation or salt form. Match on the full key when you can, and fall back to the block for a candidate list you review by hand.

6. Join HMDB to your own metabolite measurements

This is the step that produces value, and the step where most annotation errors originate. Assume you have a CSV of measured metabolites from a panel. Each row has a vendor name, an optional HMDB ID, and a concentration. Name matching is unreliable, so use identifiers in this priority order. Start with the HMDB accession through the alias table, then the full InChIKey, then InChIKey block one. Fall back to curated name matching reviewed by a human.

CREATE OR REPLACE TABLE panel AS
SELECT * FROM read_csv_auto('my_panel.csv');

SELECT p.vendor_name, p.value_uM, m.accession, m.name,
       m.taxonomy_class, m.chemical_formula
FROM panel p
LEFT JOIN accession_alias a
  ON a.alias = LPAD(REPLACE(p.hmdb_id, 'HMDB', ''), 7, '0')
             . ''  -- see note on normalization below
LEFT JOIN metabolites m ON m.accession = a.accession;

In practice normalize identifiers in Python before the join, because vendor files mix HMDB0000122, HMDB00122 and bare 122. A single regex that strips the prefix, zero-pads to seven digits and re-prefixes removes most of the pain. Log every unmatched row rather than dropping it. Unmatched rows cluster into two groups: lipid species with vendor-specific shorthand, and xenobiotics such as drug metabolites and food compounds.

The concentration tables deserve a specific warning. HMDB’s normal and abnormal concentration records are aggregated from published studies. Units, ages, sexes and subject conditions are heterogeneous across those records. You will find the same metabolite reported in µM, mg/dL and µg/mL in different rows. Convert deliberately using the formula and molecular weight from the metabolite table, and keep the original value and unit alongside the converted one. These ranges are literature summaries, not clinical reference intervals from your laboratory, and they should never be read as a personal result. If a measured value looks far outside the published range, that is a question for a physician. It is not a conclusion you draw from a database join.

7. Add spectral data only if you are doing untargeted matching

If your data came from a targeted panel with known analytes, skip this. If you have untargeted LC-MS/MS and a list of unidentified features, the HMDB spectral archives let you do local library matching. HMDB distributes spectra in mzML for mass spectrometry and nmrML for NMR, plus MS/MS peak lists in text form.

The distinction that governs how you interpret a hit is experimental versus predicted. Experimental spectra were measured on real standards. Predicted spectra were computed in silico, they cover far more of the database, and a match against them is a weaker piece of evidence. We keep them in separate tables with an explicit evidence column, and we never let a predicted-only match be reported at the same confidence as an experimental one. Annotation practice across metabolomics databases has long had to distinguish measured from inferred content for exactly this reason 6.

Expect the predicted archive to contain hundreds of thousands of files. Do not unzip it into one flat directory on a filesystem that dislikes large directories. Stream directly from the archive with Python’s zipfile and write a single Parquet table of peak lists keyed by HMDB accession, adduct and collision energy.

8. Pin the version and script the refresh

HMDB serves the newest release under a path containing current, so a rerun of the same command months later gives you a different database. Treat the download as an immutable artifact. Keep the dated directory, keep SHA256SUMS, and record the release number reported on the download page in a small MANIFEST.json next to the data.

Then make the whole pipeline one command, so a refresh is cheap and reproducible.

#!/usr/bin/env bash
set -euo pipefail
REL=$(date +%Y-%m)
mkdir -p "hmdb/$REL" && cd "hmdb/$REL"
curl -fL --retry 5 -C - -o hmdb_metabolites.zip \
  https://hmdb.ca/system/downloads/current/hmdb_metabolites.zip
shasum -a 256 hmdb_metabolites.zip > SHA256SUMS
unzip -o hmdb_metabolites.zip
python ../../hmdb_to_parquet.py
duckdb hmdb.duckdb < ../../hmdb.sql
duckdb hmdb.duckdb -c "SELECT COUNT(*) AS n_metabolites FROM metabolites;"

When you regenerate annotations, diff the new accession set against the old one. Records get merged and reclassified between releases, and a metabolite that silently changed its chemical taxonomy will change any downstream grouping you built on it.

Common problems

Zero rows parsed. Almost always the namespace. Confirm your tag string is {http://www.hmdb.ca}metabolite and not metabolite.

Memory climbing until the process is killed. You cleared the element but did not delete preceding siblings from the parent. Add the while el.getprevious() is not None loop.

XMLSyntaxError partway through the file. The download truncated. Compare the unzipped file size to the value on the download page and check that the file ends with a closing </hmdb> tag using tail -c 100.

Every InChIKey join fails. Vendor files often carry lowercase or whitespace-padded keys. Uppercase and strip both sides before matching, and verify the key is 27 characters with two hyphens.

Duplicate rows after joining. A metabolite can appear under both a primary and a secondary accession in your alias table, and a vendor file can list the same analyte twice under different names. Deduplicate on (accession, vendor_name) and count before and after.

Unit chaos in the concentration tables. Do not assume µM. Read concentration_units per row, convert explicitly, and discard rows where the unit is missing or free text you cannot parse.

Lipids that will not match. Vendor lipid shorthand does not map cleanly onto HMDB names or structures. Treat lipid annotation as a separate problem with a lipid-specific resource, and leave those features unannotated rather than guessing.

Accessions that exist nowhere. Some IDs in older publications were withdrawn during curation. The alias table will tell you when an ID was folded into another record, and silence there means the entry is gone.

Footnotes

  1. D. S. Wishart, C. Knox, A. C. Guo, et al. HMDB: a knowledgebase for the human metabolome. Nucleic Acids Research, 2009. https://doi.org/10.1093/nar/gkn810 ↩

  2. Marcella Attimonelli, Matteo Accetturo, Monica Santamaria, et al. HmtDB, a Human Mitochondrial Genomic Resource Based on Variability Studies Supporting Population Genetics and Biomedical Research. BMC Bioinformatics, 2005. https://doi.org/10.1186/1471-2105-6-s4-s4 ↩

  3. Francesco Rubino, Roberta Piredda, Francesco Maria Calabrese, et al. HmtDB, a genomic resource for mitochondrion-based human variability studies. Nucleic Acids Research, 2011. https://doi.org/10.1093/nar/gkr1086 ↩

  4. Shaojing Wang, Yuan Yun, Xuefeng Tian, et al. HMDB: A curated database of genes involved in hydrocarbon monooxygenation reaction with homologous genes as background. Journal of Hazardous Materials, 2023. https://doi.org/10.1016/j.jhazmat.2023.132397 ↩

  5. Ian J. Forsythe, David S. Wishart. Exploring Human Metabolites Using the Human Metabolome Database. Current Protocols in Bioinformatics, 2009. https://doi.org/10.1002/0471250953.bi1408s25 ↩

  6. Eden P. Go. Database Resources in Metabolomics: An Overview. Journal of Neuroimmune Pharmacology, 2009. https://doi.org/10.1007/s11481-009-9157-3 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.