Protein language models (pLMs) learn from raw amino-acid sequences and turn each protein into a numerical “embedding.” Unlike AlphaFold (which is trained to predict 3D structure), pLMs are trained only on sequence patterns, then reused for tasks like fast homology search, function hinting, or variant triage.

The appeal of pLMs is that they might:

  • Identify remote homology
  • Assist in function annotation, especially for uncharacterized proteins
  • Predict mutation effects
  • Serve as general-purpose inputs for fold prediction, domain classification, etc.

This study benchmarks 14 pLMs to evaluate how well they capture biological similarity along three axes:

  • Sequence
  • Structure
  • Function

For each axis, the authors compare distances between embeddings to a “ground truth” similarity metric (see below). They test whether models reflect these similarities in two ways:

  • Inherent information: Does the raw embedding distance between two proteins correlate with their similarity?
  • Extractable information: Can a small model trained on the embeddings predict the similarity score?

Scientific Insight

This paper provides a systematic, well-designed benchmark for protein language models, useful for those developing or deploying these models computationally.

But it defines “success” in terms of alignment with thorny labels, which is a limitation of the field at the moment. See PDF below.

With that caveat, the results are compelling:

Out-of-the-box, small models often perform as well as large ones.

  • “Size-performance paradox”: model size doesn’t guarantee better embedding quality for biological similarity.

Larger models encode more information, but it’s hidden.

  • You need to train a model on the embeddings to “extract” biological signal.

Task-specific models don’t generalize.

  • Fine-tuning a model for one problem (e.g., enzyme function) distorts the embedding space and reduces general usefulness.

Leadership Angle

Bigger isn’t automatically better (we’re seeing this with genomic models as well). But also be mindful of what you are benchmarking to, and its relevance to what you care about.

AI ∩ Bio

Illustration of scientific paper with the title "AI ∩ Bio: Reading the Revolution"

More in this series

  1. ChatNT: The future of biological assistants—or a mirage in a lab coat?
  2. GET: A Foundation Model for Transcription, Still Between Promise and Proof
  3. X-Atlas/Orion: Your Model is Only as Good as Your Training Data
  4. From Better Models to Better Questions: A Pathology AI Rethink
  5. The Model Isn’t the Magic: How Cytoland shows that domain expertise—not just deep learning—is what makes AI in biology work.
  6. Boltz-2: How much can 3D structure really tell us about molecular binding energetics?
  7. Investigating the volume and diversity of data needed for generalizable antibody–antigen ΔΔG prediction
  8. Beyond Binding: Rethinking Drug Design in the Age of AI and Structural Biology
  9. Testing the Physics Beneath the Predictions Beyond RMSD: What AlphaFold3 Really Understands
  10. Hype, Hurdles, and Hepatotoxicity: A Bold Step for AI-Designed Drugs, But Still Miles to Go
  11. When Complexity Misleads
  12. What Are Genomic Transformers Actually Learning?
  13. mRNABench and the Future of AI in Biology: Why Domain Knowledge Wins
  14. OncoGAN Creates Synthetic Cancer Genomes, Opening New Paths for Privacy-Preserving Precision Oncology
  15. Readable Rules, Testable Models: A New Grammar for Virtual Cells
  16. Multimodal CustOmics: Fusing Pathology Images and Tumor Genomics for Next-Gen Cancer Diagnostics
  17. Beyond Perturbation Simulations: PDGrapher Shows a Faster Way to Identify Actionable Targets
  18. Can AI design epigenetic anti-aging strategies?
  19. What happens when physicians use GPT-4 for diagnosis
  20. Can generative AI predict emergent phenomena?
  21. DeepSomatic and the question of how AI learns from itself
  22. Toward Mechanism-Centric Interpretability in Genomic Machine Learning
  23. From AlphaEvolve to DeepEvolve: What We’re Learning About Machine-Led Scientific Discovery
  24. Kosmos and the Culture of Discovery
  25. From Embeddings to Insight
  26. From Prediction to Explanation: How BIOREASON Reframes Genomic AI as a Reasoning Problem
  27. AI Models Need Better Truth—Platinum Pedigree Shows How
  28. From Evolutionary Intolerance to Clinical Insight: What popEVE Teaches Us About Missense Variants
  29. Interpretable Latent Spaces, Messy Biology: What AUTOENCODIX Teaches Us About Autoencoders in the Wild
  30. The Gene Ontology Knowledgebase in 2026

Mentorship Angle

Let pLMs speed the front end of discovery, then make the science real.

  • Start with a testable claim in plain language—e.g., “Nearest neighbors in embedding space enrich for shared catalytic residues better than a sequence-identity cutoff at the same recall.”
  • Pre-register thresholds; include hard negatives (same fold, different function; low identity, same mechanism) & use controls (shuffle labels, etc)
  • Report where it fails—disordered regions, multi-domain proteins, complexes, and say why you think it failed
  • Document assumptions (databases, structure confidence, training leakage)

The win isn’t a pretty plot; the win is turning speed into better experiments that change what you do at the bench.

View Fullscreen
Skip to PDF content