Kristin Gleitsman

Leadership in Biotech

Tag: benchmarks

Illustration of a desk with a figure from a recent paper about Platinum Pedigree

AI Models Need Better Truth—Platinum Pedigree Shows How

By

On November 29, 2025

In AI ∩ Bio

When I was in high school, I was obsessed with genetics. The Human Genome Project was in full swing, and it felt like the future was being written in real time. I told a family friend I wanted to become a geneticist. He smiled and said, “My cousin is at the NIH. They’ll finish the human genome before you finish college, so I wouldn’t bother.”

The project wrapped in 2003. But papers like this remind me how wrong that prediction was. Even after “finishing” the genome, we’re still uncovering what accuracy, completeness, and truth really mean.

The new Platinum Pedigree study pushes that frontier again.

Scientific Insight

This work builds one of the most comprehensive germline variant benchmarks to date, deep long-read sequencing across a 10-member family, combined with Mendelian logic.

By integrating PacBio HiFi, Oxford Nanopore Technologies, and Illumina and testing every variant against inheritance patterns, the authors defined 2.77 Gb of high-confidence genome (~200 Mb beyond prior benchmarks), including repeats, segmental duplications, and low-mappability regions.

The key innovation is biological grounding.

Each child inherits one haplotype from each parent; variants that obey those segregation patterns are kept, and those that don’t are removed. This yielded ~4.7M SNVs, 768k indels, 537k tandem repeats, and 24k structural variants as pedigree-consistent truth.

When DeepVariant was retrained on this truth set, error rates dropped by ~34% across challenging classes, especially indels and tandem repeats.

Better labels → better models.

Leadership Angle

For diagnostics leaders, this signals where the field is heading: stronger evidence standards, clearer definitions of “truth,” and biologically informed benchmarks rather than technology-constrained heuristics.

This strategy of combining multiple sequencing technologies and adjudicating discrepancies with inheritance is exactly how robust systems are built in uncertain environments.

It mirrors what clinical diagnostics now requires: pipelines that perform not just in easy regions, but in messy, clinically meaningful ones.

And it underscores a central lesson in AI-enabled diagnostics: your model is only as good as the ground truth you train it on.

The regions that are currently messy and difficult to map: that’s where new breakthroughs in understanding will occur.

AI ∩ Bio

Illustration of scientific paper with the title "AI ∩ Bio: Reading the Revolution"

Mentorship Angle

For early-career scientists, the lesson is craftsmanship. This paper doesn’t debut a flashy algorithm; it elevates the foundations. It asks simple but profound questions: Did this variant follow the rules of inheritance? If not, are we sure it’s real?

Your technical tools matter, but your willingness to interrogate assumptions matters more. If you want to build a meaningful career in genetics in this age of AI, stay curious about the scaffolding beneath the science.

Breakthroughs often start there.

View Fullscreen
Skip to PDF content
Illustration of a desk with a figure from a recent paper about the pLM

From Embeddings to Insight

By

On November 15, 2025

In AI ∩ Bio

Protein language models (pLMs) learn from raw amino-acid sequences and turn each protein into a numerical “embedding.” Unlike AlphaFold (which is trained to predict 3D structure), pLMs are trained only on sequence patterns, then reused for tasks like fast homology search, function hinting, or variant triage.

The appeal of pLMs is that they might:

  • Identify remote homology
  • Assist in function annotation, especially for uncharacterized proteins
  • Predict mutation effects
  • Serve as general-purpose inputs for fold prediction, domain classification, etc.

This study benchmarks 14 pLMs to evaluate how well they capture biological similarity along three axes:

  • Sequence
  • Structure
  • Function

For each axis, the authors compare distances between embeddings to a “ground truth” similarity metric (see below). They test whether models reflect these similarities in two ways:

  • Inherent information: Does the raw embedding distance between two proteins correlate with their similarity?
  • Extractable information: Can a small model trained on the embeddings predict the similarity score?

Scientific Insight

This paper provides a systematic, well-designed benchmark for protein language models, useful for those developing or deploying these models computationally.

But it defines “success” in terms of alignment with thorny labels, which is a limitation of the field at the moment. See PDF below.

With that caveat, the results are compelling:

Out-of-the-box, small models often perform as well as large ones.

  • “Size-performance paradox”: model size doesn’t guarantee better embedding quality for biological similarity.

Larger models encode more information, but it’s hidden.

  • You need to train a model on the embeddings to “extract” biological signal.

Task-specific models don’t generalize.

  • Fine-tuning a model for one problem (e.g., enzyme function) distorts the embedding space and reduces general usefulness.

Leadership Angle

Bigger isn’t automatically better (we’re seeing this with genomic models as well). But also be mindful of what you are benchmarking to, and its relevance to what you care about.

AI ∩ Bio

Illustration of scientific paper with the title "AI ∩ Bio: Reading the Revolution"

Mentorship Angle

Let pLMs speed the front end of discovery, then make the science real.

  • Start with a testable claim in plain language—e.g., “Nearest neighbors in embedding space enrich for shared catalytic residues better than a sequence-identity cutoff at the same recall.”
  • Pre-register thresholds; include hard negatives (same fold, different function; low identity, same mechanism) & use controls (shuffle labels, etc)
  • Report where it fails—disordered regions, multi-domain proteins, complexes, and say why you think it failed
  • Document assumptions (databases, structure confidence, training leakage)

The win isn’t a pretty plot; the win is turning speed into better experiments that change what you do at the bench.

View Fullscreen
Skip to PDF content
Illustration of a desk with a figure from a recent paper about DeepSomatic

DeepSomatic and the question of how AI learns from itself

By

On October 18, 2025

In AI ∩ Bio

DeepSomatic, published this month in Nature Biotechnology, represents a milestone for cancer genomics: a deep-learning method that detects somatic small variants across both short- and long-read sequencing data. Built on Google’s DeepVariant framework, it bridges Illumina, PacBio HiFi, and Oxford Nanopore datasets and introduces CASTLE, a new multi-platform benchmark of six tumor–normal cell lines made openly available to the community. For anyone working in precision oncology, the technical ambition here is remarkable: one model spanning technologies, sample types, and variant classes.

Scientific Insight

DeepSomatic converts paired tumor–normal reads into tensor “images” that feed a convolutional neural network capable of distinguishing somatic, germline, and reference variants. The model outperformed leading tools such as Strelka2 and ClairS across variant types and variant allele frequencies, and it maintained accuracy across multiple sequencing chemistries. Beyond its raw performance, the CASTLE dataset fills a major gap in the field: creating a real benchmark for long-read somatic variant detection where none previously existed.

Scientific Rigor Note

Like many GenAI systems, DeepSomatic may fall pray to non-obvious data leakage, and would benefit from more explainability. Some of its evaluation data overlap with the model’s own training inputs, raising the risk of circular benchmarking bias, and the study offers little insight into why the network makes its calls.

AI ∩ Bio

Illustration of scientific paper with the title "AI ∩ Bio: Reading the Revolution"

Leadership & Mentorship Reflection

Building trustworthy AI in medicine requires independent data, transparent reasoning, and humility about limitations that are baked into how these models work.

For early-career scientists, this paper is a case study in responsible ambition: innovate boldly, share your data openly, and interrogate your own benchmarks. AI or not, progress comes from rigorous, open science that understands its own limitations.

Illustration of a desk with a figure from a recent paper about mRNABench

mRNABench and the Future of AI in Biology: Why Domain Knowledge Wins

By

On August 23, 2025

In AI ∩ Bio

This week’s paper, “MRNABENCH: A curated benchmark for mature mRNA property and function prediction,” introduces a benchmarking framework for evaluating whether foundation models are truly learning features of RNA biology, specifically as it relates to mRNA. Messenger RNA is one of the most information-dense molecules in biology, carrying not only the coding sequence but also a layered regulatory grammar across UTRs, splice isoforms, and motifs (we won’t get into modifications today, but there’s that too). These features govern stability, localization, and translation efficiency, dimensions central to both basic biology and therapeutic design.

Scientific Insight

What makes this work stand out is its clear demonstration that models designed with biological principles in mind rival or exceed massive models in many tasks, highlighting biologically grounded design as equally important as scale. The authors show that models aligned with transcript biology can match or even surpass billion-parameter models on key benchmarks, delivering strong results at far less computational cost. Equally important, their rigorous approach to data splitting (random, k-mer, and homology-based) reveals a common blind spot in genomic machine learning, where models often appear to generalize but are simply re-identifying homologous sequences. In other words, success was linked to respecting the rules of molecular biochemistry, not just piling on more unlabeled data.

Leadership Angle

For leaders in diagnostics and therapeutics, this work is a powerful reminder: scaling isn’t everything. In an era where compute budgets are skyrocketing, the true differentiator may be how well we integrate domain knowledge into AI design. Frameworks like mRNABench help us separate hype from genuine progress, ensuring that models capture biologically meaningful signals, an essential step toward reliable applications in biology and therapeutics.

AI ∩ Bio

Illustration of scientific paper with the title "AI ∩ Bio: Reading the Revolution"

Mentorship Angle

For early-career scientists, the takeaway is clear: don’t lose sight of the biology. It’s tempting to chase ever-larger models or datasets, but this paper shows the biggest leaps often come from framing the right questions and aligning methods with molecular reality. Building rigorous standards, designing smarter architectures, and spotting blind spots in evaluation are contributions that will shape the field for years to come. If you’re wondering how to make your mark, focus on creating the kind of cross-domain exchange where the biological questions and scientific rigor are foundational to your approach, not an afterthought.

Powered by WordPress & Theme by Anders Norén