MIRAGE

Co-folding models such as Boltz-2 and Nesso-1 report protein-ligand affinity correlations near Pearson 0.6 and are proposed as fast surrogates for free-energy perturbation. That single number cannot separate a model that has learned binding physics from one that recognizes protein families it has seen many times in the PDB.

The distinction decides programs. A medicinal chemistry campaign succeeds or fails on novel targets, the ones least represented in public structural and affinity data. So MIRAGE asks the question the overall correlation hides: how does accuracy change as a target's family becomes less familiar?

The central result is an interaction, not a ranking. Frontier co-folders depend strongly on historical public family support. Family-disjoint shallow controls do not. On novel families the practical ranking inverts, and on one external low-support target neither co-folder convincingly beats molecular weight.

We call this redundancy-driven inflation rather than leakage: proprietary training sets cannot be reconstructed, so family density stays a proxy for exposure rather than evidence of it, and the design does not separate memorisation from other family-correlated difficulty.

Family support, S_f

For a target in family f, the inclusive number of PDBbind structures in the same MMseqs2 30% sequence-identity cluster, the evaluation complex included, so a singleton family has S_f = 1. It is historical public support through 2019: every evaluation complex is a 1982–2019 PDBbind deposition. Binned into {1, 2–5, 6–20, 21–80, 81–300, 301+}.

Matched strata

Within each support bin, Pearson r is computed over a matched pK window of [4.5, 8.0], holding label spread, and therefore attainable correlation, approximately constant across bins.

Family generalization gap, G_m

G_m = r(S_f ≥ 301) − r(S_f = 1)The change in accuracy from singleton to well-supported families. A positive G_m means performance increased with family support. Estimated with a two-level bootstrap: resample families with replacement, then resample ligands within each family, so the interval reflects both family-level and ligand-level variability.

Controls

The honest counterpart to a frozen co-folder, whose test families sit inside its training window, is a model that cannot have seen the test family. A random-forest QSAR model and a ligand-kNN baseline are evaluated under 5-fold family-disjoint cross-validation (GroupKFold on family id), so every prediction is on a held-out family.

The chart at the top of this page, as numbers. Row maxima in bold: the co-folders peak at the most redundant bin, the controls at the most novel one.

Table 2. Matched-pK Pearson r by protein-family support.

The dependence survives the measured covariates. Regressing per-target absolute error on log₁₀ S_f together with ligand similarity, protein length, publication year, within-family affinity variance, assay type, and the affinity regime, the family-support coefficient stays large and significant: Nesso-1 −0.130 (t = −5.1), Boltz-2 −0.152 (t = −3.4). Standard errors are CR1 cluster-robust, clustered on protein family (1,410 and 373 clusters), matching the dependence structure the two-level bootstrap assumes elsewhere.

Nor is the gradient exact-target repetition. Adding a repetition indicator leaves the coefficient unchanged (−0.130 → −0.135, t = −5.2) with the indicator insignificant, and restricting to targets whose sequence appears nowhere else in the corpus leaves the gap undiminished: +0.49 [+0.23, +0.74] against +0.45 unrestricted, with every control still flat. Repetition amplifies the effect rather than creating it.

Two memorization channels are available: co-folders can exploit protein-family familiarity, and ligand models can exploit chemical similarity. Matching affinity distributions alone does not separate them, so MIRAGE stratifies by both at once.

Table 3. Nesso-1 Pearson r, matched-pK. Accuracy climbs down the family axis, from about 0.1 to about 0.5, and is flat across the ligand axis. The memorization channel for co-folders is protein-family familiarity, not ligand chemistry.

A predictor that outputs nothing but the mean pK of the target's family scores 0.564 under random splitting, and approximately zero under family-disjoint splitting by construction. That is close to the co-folders' pooled 0.588. Much of what a random split measures is recoverable protein-family identity rather than transferable modelling of protein-ligand binding. This tests one ligand statistic, nearest-neighbour ECFP4 Tanimoto: scaffold identity, ligand size and congeneric structure remain untested, so MIRAGE does not close the ligand channel.

A model given only the protein, blind to the ligand and so unable in principle to know the affinity, still reaches 0.64 under random splitting. It collapses under family-disjoint splitting. Each row below is one feature set, from its random-split score to its family-disjoint score.

Table 4. Random forest, Pearson r. Gap column is the drop from random to family-disjoint splitting.

The external arm asks whether the low-support result reproduces on a newly released ligand series. The target is the OpenBind EV-A71 2A protease, which has two prior public family structures, so it sits at the low-support end of the curve; what post-dates every documented cutoff is the ligand series and its affinity measurements, not the protein fold. Because no evaluated model discloses an affinity-data cutoff, this is a zero-shot temporal evaluation, not a prospective one.

Ranking 643 compounds against it, molecular weight sits second from the top. Nesso-1 exceeds it only marginally, and Boltz-2, Gnina, smina, AEV-PLIG, clogp and AQ-Affinity all fall below it.

Table 5. Spearman ρ over the 643 compounds with complete predictions, and ρ after partialling out molecular weight (right column). Five of the seven methods are negatively informative once size is removed: their apparent ranking skill on this target was size tracking.

Two external sources agree with this reading. Nesso-1's own report states its advantage over molecular weight on this target is not statistically convincing, and the OpenBind release itself flags molecular weight as a strong baseline for the campaign. It is reported here as one external target, not population-level proof: a single congeneric series cannot carry that weight.

The gradient is not a property of co-folding, or of model size. Re-scoring the crystal pose separates two regimes: a classical empirical scoring function is weak overall and shows no improvement with family support, while a CNN score trained on PDBbind is far stronger and carries the same dependence as the co-folders.

Table 6. Docking scoring functions on the affinity set. smina's slope is significantly positive, the sign molecular weight also shows: error rises slightly as families become better represented. We state that as measured rather than as flatness, and offer no mechanism for the sign.

smina and Vina are regression-fitted empirical functions, not free-energy calculations, which is why this contrast is not described as physics versus learned. Two scorers do not establish a law about every learned function, and the paper does not state one.

The same family axis applies to geometry. Crystal ligands are redocked into 50 singleton and 50 redundant targets, scored by symmetry-corrected RMSD, with success at RMSD < 2 Å and a missing pose counted as a failure. Pose-G_m is redundant minus novel success.

Table 7. Redocking, n = 50 per bin. The first three engines are given the crystal receptor, where smina sets a +0.23 difficulty reference for how much harder the novel targets are; the co-folders build the complex de novo from strictly less information, so their gaps are not that reference plus a memorisation excess and nothing is subtracted.

The controlled comparison for a co-folder is therefore within-model. With weights, architecture and target set fixed, a ColabFold MSA raises Boltz-2's novel-family success from 2% to 46% while redundant success stays at exactly 66%, collapsing pose-G_m from +0.64 to +0.20. The effect is one-directional: 22 of 50 novel complexes flip from failure to success and none the reverse, and both settings posed all 100 targets. Chai-1 gains on novel families too (22% to 33%) but loses on redundant ones and suffers run attrition, so it counts as partial directional replication.

Supplying evolutionary context therefore removes much of the pose deficit associated with low public family support. That deficit is localisable and at least partly a missing-input problem, not an irreducible one. Family novelty here is defined on PDBbind structural support while ColabFold retrieves from far larger sequence databases, which is precisely why the rescue is possible.

Chai-1 and ESMFold2 predict structure and an interface confidence (ipTM), not affinity. Scoring ipTM as an affinity proxy is a weaker signal than a trained affinity head, and the two proxies differ instructively.

Table 8. Interface ipTM scored as an affinity proxy.

Chai-1's ipTM carries the same family-support dependence, rising from 0.155 on novel families to 0.461 on redundant ones (slope t = −3.4). Since Chai-1 has no affinity head, this is consistent with the dependence residing in the learned structural representation rather than only in a regression head fitted to affinity labels, though ipTM is a confidence score and is itself responsive to difficulty. ESMFold2's ipTM is a weak and noisy proxy with no significant dependence (t = −0.5), so the effect is not universal across confidence proxies; that arm is thin and nothing further is drawn from it.

The thesis is not specific to MIRAGE's own dataset. ATOM3D LBA is the field's standard structure-based affinity-generalization benchmark: PDBbind-refined complexes split by 30% protein sequence identity, on which recent geometric deep nets report large gains. No published LBA30 result reports a ligand-only, molecular-weight, or QSAR control. The field compares only against other structure and sequence deep nets. Those controls are supplied below.

Table 9. Pearson r on pK, full official test split (n = 490; 489 for the fingerprint-based controls). Deep-net numbers are from the literature; the three ligand-only controls in bold are ours and, to our knowledge, have not previously been reported.

Two conclusions follow, and both are stated. First, the top methods genuinely learn structural signal: IPBind exceeds the ligand-only control by roughly 0.29, which is strong evidence of predictive signal beyond a ligand descriptor. Second, and more consequential for the field, molecular weight alone scores 0.436, above ENN, near DeepDTA, and within 0.11 of 3DCNN and GNN. Small reported margins over those methods were never contextualised against a trivial descriptor, and a reader of the published leaderboard has no way to tell which entries clear it.

The same control reconciles LBA30 with MIRAGE. A 30% sequence split is a far weaker generalization test than singleton-family stratification: a ligand-only model reaches 0.440 on LBA30 but only 0.31 to 0.41 on MIRAGE novel families. So 0.73 and 0.31 do not contradict each other. They measure different stringencies, and only the second resembles the situation of a genuinely new target.

What the paper recommends anyone reporting affinity accuracy should do:

• Weight families equally, not complexes
• Report mean within-stratum correlation, not pooled-across-target correlation
• Report Pearson, Spearman, centred RMSE, and pairwise ranking accuracy
• Use a two-level bootstrap: families first, then ligands within families
• Show sensitivity to the clustering choice
• Ship a baseline suite: family-mean (target identity), molecular weight, clogp, ligand-kNN, and RF-QSAR
• Report the family gap and the novel-family number, not a single overall correlation

• The redundancy set is pre-cutoff PDBbind, and S_f is a within-corpus support proxy, not the models' exact training counts
• The temporal set is a single target with a congeneric series, and is zero-shot rather than prospective: no evaluated model discloses an affinity-data cutoff
• S_f is historical public family density, a proxy for exposure rather than evidence of training exposure, and the design does not separate memorisation from other family-correlated difficulty
• Within-exact-target ranking is not estimable on this corpus: requiring five ligands per identical sequence leaves six targets for Nesso-1 in the redundant stratum and none in the novel one
• No arm varies training-set redundancy, so the MSA result localises the pose deficit without establishing how family redundancy acts during training
• Frozen public weights are evaluated; no model is retrained on de-leaked data
• Family support at 30% identity is one axis. Pfam-based and pocket-based clustering are natural sensitivity analyses
• The confidence-proxy results use ipTM, not a trained affinity head
• AlphaFold3 has open code but request-only weights and no affinity head. It ships as a pluggable stub and is not evaluated

MIRAGE is an installable package with a Hugging Face dataset. Scoring a model means wrapping it in one function.

from mirage import run_redundancy, run_temporal

def predict(sequence: str, smiles: str) -> float:
    ...                      # your model, higher is stronger

run_redundancy(predict)      # family-support curve + G_m
run_temporal(predict)        # external low-support target

pip install mirage

# score a predictions CSV: sequence, smiles, prediction
mirage score predictions.csv --set redundancy
mirage score predictions.csv --set temporal

from datasets import load_dataset

# 18,759 complexes, annotated with family support + ligand Tanimoto
redundancy = load_dataset("DeepBioScientific/MIRAGE", "redundancy")

# 649 compounds vs. the low-support EV-A71 2A protease
temporal = load_dataset("DeepBioScientific/MIRAGE", "temporal")

Function names follow the paper. The repository documents the current CLI flags and dataset configuration names.

A CLI scores a predictions CSV instead. The public release includes all metrics (G_m, matched-pK, the multivariate regression, and the two-level bootstrap), every baseline, the dataset pipeline, and the released predictions used to reproduce each number on this page.