This week’s AI ∩ Bio: Reading the Revolution paper isn’t just a new model or a shiny dataset. It’s a reality check that cuts through superficial benchmark performance to ask harder questions about AI for antibody design.
While many models report high accuracy on test sets that closely resemble their training data, Hummer et al. go further, testing how models perform when challenged with unseen antibody–antigen complexes. This approach led to a rigorous investigation of why models fail, and what it would take to make ΔΔG prediction truly generalizable.
They introduce Graphinity, a graph neural network trained to predict changes in antibody–antigen binding affinity (ΔΔG). They rigorously stress-test the system: varying sequence identity cutoffs, benchmarking against other architectures, simulating noisy labels, and even filtering mutations by evolutionary plausibility.
The findings are clear and sobering: it’s not the model that’s the bottleneck. It’s the data: data size, data diversity, and data alignment with real-world biological utility.
a,b, Model performance with varying training plus validation dataset size
Scientific Insight
The core contribution isn’t Graphinity’s design (though it performs well). It’s the authors’ commitment to not mistaking narrow wins for general insight. They show that a 0.9 Pearson correlation can be misleading if training and test data are too similar. That scaling dataset size doesn’t help if you’re mutating the same few complexes. And posit that no model—regardless of architecture—can learn to generalize if it’s never exposed to true diversity.
Crucially, they quantify the gap: at least 90,000 experimental ΔΔG values may be needed to reach robust generalization. And they begin to chart how we might get there—by prioritizing diverse substitutions, broad antigen coverage, realistic effect sizes, and evolutionarily grounded mutations—not just alanine scans.
To R&D leaders
This paper demonstrates what it looks like to take biological realism seriously. If you’re evaluating models for therapeutic antibody design, don’t just ask about top-line metrics. Ask how performance changes when similar complexes are removed from the test set. Dig deeply into what signal the model is learning and whether that signal will hold for the next antigen.
To early-career scientists
This paper helps us see what not fooling yourself looks like. It’s a reminder that model performance only matters when grounded in data that reflects the system’s true complexity. If you want to build models that generalize, start by designing experiments that enable that.