Machine Learning for Protein Engineering
Finding better proteins for fewer measurements
We reimplemented the reference models for deep mutational scanning, then measured the thing the benchmarks leave out: how many laboratory measurements the model actually saves.
The question a benchmark score does not answer
Deep mutational scanning measures tens of thousands of variants at once. A model trained on those measurements is meant to tell you which of the millions it did not measure are worth making. The literature scores that model with a correlation. A laboratory cannot spend a correlation.
What gets reported
Pearson r = 0.98 on a held-out split.
True, reproducible, and silent about what to do next.
What a laboratory decides
Which 96 variants go on the next plate
and whether the plate after that is worth paying for.
So we ask three different questions
- If I make a hundred of these, how good are they?
- How many measurements would random picking have needed for the same haul?
- Is the model’s best pick better than the best thing I have already measured?
What we do
Reproduce the reference models honestly
Four architectures over one-hot and AAindex features, on five public DMS datasets, in PyTorch. Reproduction is the baseline, not the result.
Replace correlation with a decision
Selection metrics throughout — captured gain, recall of the true top hundred, budget to the best — every one of them normalised against a random-picking null.
Price the target before fitting it
Enrichment scores are ratios of read counts, so they carry noise no model can predict. Splitting the reads gives a hard ceiling on any achievable correlation.
Replay the campaign, and cost it
Spend a budget the way a laboratory would — seed, retrain, choose, repeat — and score every strategy on reads it never trained on.
9 plates instead of 406
On the one combinatorially complete library in the set, 864 model-guided measurements bought 78 of the pool’s true top 100. Random spending of the same budget bought 1.
Variants from the pool’s true top 100 recovered, by measurements spent (96 per plate), 50 independent repeats.
Scored on reads the model never trained on: the counts are split into two independent replicates, and the strategy is judged on the half it did not see. 50 independent repeats.
Where that gain actually comes from
Each rung adds one thing to the one below it. Read it before crediting the result to any single component — the answer is not the architecture.
What earns the money is fitting any predictive model at all inside the loop — not the loop, and not the network. The network still pays for itself: it reaches the linear model’s final result at 592 measurements rather than 864, and it cuts the rate of a badly failing campaign from 22% to 6%.
What does not survive re-measurement
The single best variant each strategy found, measured a second time on the independent replicate. This is the column that decides whether a headline number can be quoted.
The ranking holds
78 of the true top 100, confirmed on reads the model never saw. A batch claim is safe.
The champion claim does not
Re-measured, every strategy lands within a few tenths — including random.
The test also has no power: with 50 repeats the standard error of the difference is about 0.17, so failing to separate is not the same as equivalence. And 0.62 of the 1.97 drop is incurred by random picking too — it is what taking a maximum over noisy values costs anyone.
Before pricing a model, check the target exists
All five dataset files are byte-identical to the published source, with no residue mismatches, no duplicate variants and no overlap between splits. Then there is this.
What we can take on
The part we do for someone else is evaluation: not making the model bigger, but working out which of its numbers can be acted on.
Measurement noise ceilings
The highest correlation any predictor can reach against your scores — so you know whether more tuning is worth paying for.
Data audits
Read depth, pseudocount artefacts, tie structure, split overlap, and a byte comparison against the published source.
Campaign replay
What model-guided selection returns against random, additive and zero-shot baselines at equal spend.
Pricing the winner’s curse
Separating selection bias from the reported improvement, and giving the corrected value.
If you need an evaluation, we can take the work on.
An in-house campaign, a model benchmark, or a pipeline on public data. kwlee@caidesystems.com