arxiv.orgpaperFREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification
FREA is a benchmark of 751 chemical reactions labeled by expert chemists for feasibility verification, drawn from retrosynthesis model proposals, zero-yield experimental records, LLM-generated edits, and five negative-candidate generation methods. The authors show no single verifier type (LLMs prompted with the feasibility criterion, dedicated forward models, or retrosynthesis scorers) dominates across all sources, and that adding generated negative examples to training improves mean AUROC but fails to transfer to retrosynthesis proposal screening. They also release a corpus of over 14 million recorded and generated reactions to support further training and evaluation of feasibility verifiers.
What it claims
FREA is a 751-reaction benchmark (341 feasible, 410 infeasible). Two chemists labelled every reaction against one explicit feasibility criterion, without seeing which source it came from. The headline finding is that the best verifier depends on the source:
- LLMs given only the criterion in the prompt have the best mean balanced accuracy. Claude Opus 5 scores 68.0.
- Forward models do best on retrosynthesis proposals. At the evaluated operating points, though, they reject most feasible LLM edits of recorded reactions.
- Forward models fall below chance on one subset: separating infeasible alternative disconnections (the New Center method) from feasible references. Chemformer and BARTSmiles score AUROC 26.2 and 32.5.
The labels also show that a reaction's origin does not settle its feasibility:
- 79 of 93 zero-yield experimental records were judged feasible.
- 33 of 289 generated "negatives" were judged feasible.
How it works
Task. Verifiers see only reactants and products, with no conditions. The product list need not be complete. A reaction counts as feasible only if it meets two conditions:
- Completeness. Every substantive reactant is listed. Auxiliary components can be assumed: catalysts, solvents, bases, reducing agents and water. For example, a Suzuki coupling without its organoboron partner is incomplete.
- Reactivity. Some reasonable conditions give the target products in isolable form in a single step or a one-pot operation. The target need not be the major product.
The labels do not predict yield.
Sources (feasible/infeasible):
| Source | Origin | Feasible/infeasible |
|---|---|---|
| Retrosynthesis proposals (RP) | Eight single-step models (template, semi-template and template-free) on ORDerly targets | 197/85 |
| Zero-yield records (ZR) | ORDerly reactions with a reported yield of zero | 79/14 |
| LLM perturbations (LP) | GPT-5.4 and Claude Opus 4.6 told to make recorded reactions infeasible but superficially coherent | 32/55 |
| Generated reactions (GR) | Five generation methods, sampled equally | 33/256 |
Generation methods. Three methods change the reactants and keep the product:
- Random Addition adds atoms at reactant reaction centres.
- Random Replacement swaps in a similar reactant set from another recorded reaction.
- New Center picks a different product disconnection and completes the fragments into reactants.
Two methods change the product and keep the reactants:
- Forward Template applies an alternative forward template.
- Elementary Template composes elementary reaction steps.
The same methods applied to ORDerly produce the 14M-reaction training corpus. No benchmark reaction appears in it.
Forward-model scoring uses one of two rules:
- Top-10 inclusion: accept if the target appears in the 10 highest-ranked predictions after canonicalization.
- Mean per-token log-likelihood (LL) of the target.
Evidence
Results from Table 2:
| Comparison | RP AUROC | GR AUROC | Four-source mean AUROC |
|---|---|---|---|
| BARTSmiles (LL), recorded reactions only | 78.9 (best on RP) | 66.9 | 72.8 |
| BARTSmiles (LL+neg.), continued with a loss favouring recorded over generated negatives | 77.0 | 73.5 | 75.2 |
So adding negatives helps on generated candidates and on average, but costs accuracy on the retrosynthesis proposals a planner actually screens.
Other results from the same table:
- Balanced accuracy, four-source mean:
- Claude Opus 5: 68.0
- BARTSmiles top-10: 65.9
- GPT-6 Astra: 65.7
- Chemformer LL: 63.9
- RP balanced accuracy: BARTSmiles top-10 is best at 73.1.
- LP: Claude Opus 5 has the best balanced accuracy at 76.1. A descriptor GBDT using 58 chemical descriptors has the best AUROC at 75.4.
- GR: forward models are close to chance on balanced accuracy, e.g. Chemformer top-10 at 50.1.
- Discriminative model: BARTSmiles trained to output a feasibility score directly is the weakest trained-on-negatives system, at 69.7 mean AUROC.
Varying the generation method, the method with the largest gain on generated candidates also made proposal screening worse. The training comparisons are described as matched and pre-registered.
How to use it
Both datasets are on Hugging Face:
- Benchmark:
https://huggingface.co/datasets/FREA-benchmark/FREA-benchmark - Corpus:
https://huggingface.co/datasets/FREA-benchmark/FREA-corpus
The benchmark release includes the annotators' recorded failure reasons and notes where available. Report results per source, not only the mean, because the rankings flip between sources.
Limits
- Small, unbalanced subsets. ZR has only 14 infeasible examples. GR has only 33 feasible ones. Per-source metrics on these are noisy.
- Small annotator pool. There are two annotators: a pharmacy faculty member and a chemistry postdoc. Agreement statistics are in Section B, which the fetched text does not include.
- Fixed operating points. Balanced-accuracy results depend on the evaluated thresholds.
- Truncated page. The fetched text stops in the evaluation protocol. The per-method transfer numbers (Section 7) and the per-model and per-method breakdown (Table C.1) were not available here.
- No licence for the benchmark or corpus is stated in the fetched text. Check it before you adopt either.
Organisations named here
- FREA
Techniques named here
- Reaction feasibility verification
- Negative candidate generation
- Forward model verification
- LLM-based feasibility judgment
Related in evals
