BioEVAL: Testing AI for Bioengineering with Images

BioEVAL: Testing AI for Bioengineering with Images

Introduction

AI is getting good at biomedical text, but bioengineering is not just reading papers—it’s deciding what to do next in real experiments. That’s the motivation behind a new benchmark called BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institution research effort designed to test large language and multimodal models on PhD-level bioengineering tasks. This work is reported in new research from the original paper.

What’s refreshing about BioEVAL is the focus. Instead of only measuring “can the model recall facts?”, the benchmark measures whether models can do research-like reasoning: interpreting experimental scenarios, synthesizing scientific literature into publication-style abstracts, and even understanding visual experimental evidence like gels, plots, and microscopy images. The benchmark spans 11 major bioengineering subfields plus uncategorized items, built by a consortium of 22 research groups across North America, Europe, and Asia.

BioEVAL includes 608 evaluation items total: 380 multiple-choice questions (MCQs) (359 retained after an audit), 218 literature synthesis tasks, and 10 multimodal problems for interpreting experimental images. The researchers also evaluated a mix of models—frontier cloud-scale systems (like ChatGPT, Gemini, and Grok) and smaller models you can run on consumer-grade GPUs—to understand how capability changes with deployment constraints.

Why This Matters

Here’s the expert take: the gap BioEVAL targets is the one that keeps showing up whenever people try to “plug AI into the lab.” A model might sound confident on a protocol question, but experiments punish shallow reasoning. In practice, bioengineering is a constant loop of hypothesis → experiment → evidence → troubleshooting. BioEVAL tests whether models can survive that loop at least in a controlled, benchmarked form.

This is especially relevant right now because many teams are already using general-purpose chatbots for literature digestion, experimental planning, or “what does this plot mean?” guesswork. BioEVAL doesn’t just ask whether the model can answer—it checks whether the answer is backed by explanation quality, and whether performance collapses when questions shift from conceptual principles to messy experimental interpretation. In other words, it’s closer to how AI would actually be used in daily work.

A concrete scenario: imagine you’re setting up a fed-batch run and you want AI assistance in deciding whether an inoculum culture looks ready, or whether a qPCR curve implies a meaningful fold difference. BioEVAL’s multimodal questions include exactly that kind of structured interpretation (e.g., qPCR amplification curves, SDS–PAGE gels, growth curves, sequencing traces). In the pilot multimodal benchmark, top-performing models reached 80% accuracy on 8 out of 10 questions—not “magic,” but clearly non-trivial. That’s the kind of reliability signal teams can use today to decide where AI should help versus where a human must take over.

Compared to earlier AI benchmarking in biology and medicine, BioEVAL builds on a key limitation in many datasets: they often emphasize text-only Q&A or diagnosis-like reasoning rather than bioengineering experimental reasoning and multimodal evidence. The paper positions BioEVAL as complementary to efforts like LAB-Bench and BioProBench (procedural biology reasoning) and earlier scientific QA benchmarks. BioEVAL adds a bioengineering-subfield structure, integrates literature synthesis, and pilots real multimodal experimental interpretation—plus it includes expert audits for benchmark quality control. That audit piece matters more than people expect.

What the Researchers Actually Measured: accuracy, explanation alignment, and multimodal evidence

BioEVAL is built around three evaluation modes—each meant to approximate a different “research skill”:

1. MCQs (multiple-choice questions): PhD-level bioengineering questions written from real experimental scenarios. The model answers an option, then generates explanations for all four options, which are compared against expert rationales.
2. Literature synthesis (L-Syn): models read full text with abstracts withheld, then generate a publication-style abstract. The output is scored against the original author abstract using semantic similarity.
3. Multimodal reasoning (MRQs): models interpret experimental visuals (plots/images) alongside domain reasoning—framed as MCQs with images.

A key detail: the benchmark isn’t “just accuracy.” For MCQs and multimodal tasks, BioEVAL also measures semantic agreement between model-generated explanations and expert-authored explanations. For literature synthesis, it measures semantic agreement between generated abstracts and reference abstracts.

The benchmark design: 608 items across 11 bioengineering subfields (and audits that remove bad questions)

BioEVAL spans 11 categorized bioengineering subfields plus uncategorized items, with items authored by experts from 22 research groups. After centralized quality control, the benchmark totals:

• 380 MCQs authored, with 359 retained after audit
• 218 literature synthesis tasks
• 10 multimodal reasoning problems

MCQ audit: 21 questions withheld to protect benchmark integrity

The paper describes a careful quality control process after model evaluation: the researchers pool the highest-accuracy and lowest-accuracy MCQs into a 40-item cross-group audit, then run a blinded consensus review with domain experts. Items recommended for revision/removal are withheld from scoring.
Result: 21 questions flagged for revision or removal were withheld, leaving 359 MCQs for reported results. Reported MCQ accuracy is computed only on those retained items.

This design choice is a big deal because it prevents a common failure mode in benchmarks: if an item is ambiguous or has a flawed answer key, it can make models look worse (or better) for reasons unrelated to capability. The audit also revealed that many troublesome items weren’t “too hard”—they were often problematic due to ambiguous phrasing, unclear superlatives, or multiple defensible answers.

How well do models do? Cloud vs edge, conceptual vs experimental, and subfield “unevenness”

BioEVAL evaluates a panel of 21 contemporary LLMs for MCQs and a subset for other tasks. It also splits models by deployment strategy:

• Cloud-scale (frontier) models
• Edge-deployable open-weight models that can run on consumer-grade GPUs

Model performance on MCQs: top systems hit 90%, edge models still strong

On the 359 retained MCQs, frontier models did best overall. The paper reports:

• Highest accuracy: Gemini-2.5-Pro at 90%
• Several other frontier systems at above 85%
• Edge-deployable models show meaningful performance too:
  • GPT-oss-20B reaches 85%
  • Qwen-3.5-9B reaches 80%

Conceptual vs experimental questions: experimental reasoning is harder

BioEVAL tags MCQs as experimental vs conceptual. Across most models, the pattern is consistent:

• Experimental MCQs score lower than conceptual MCQs
• This suggests models are more reliable with principles than with experiment-driven interpretation (design, troubleshooting, reading evidence, etc.)

That gap is exactly what you’d expect in real bioengineering: knowing what a mechanism is doesn’t automatically tell you what happens when you run a protocol and get imperfect data.

Explanation alignment: accuracy isn’t the whole story

BioEVAL also compares model-generated explanations against expert rationales using a semantic similarity metric. Here’s an important nuance from the paper:

• Explanation similarity distributions generally follow accuracy trends, but
• many models show intermediate explanation agreement, even when answers are correct.

The paper notes that in the explanation similarity distributions, the most accurate models tend to cluster around a similarity score of ~0.5, while some models show higher variance. A striking example: some Qwen models have relatively good answer accuracy but lower mean explanation alignment and more variability.

So: models may “pick the right option” while generating explanations that don’t match expert reasoning closely. That matters if you’re using AI not just to answer, but to justify decisions in a lab setting.

Subfield-level results: performance isn’t uniform

Perhaps the most practical outcome of BioEVAL is the “heatmap” idea: where models are strong vs where they struggle.

• Consistently strong subfields for MCQs include:
  • Diagnostics, Biosensing, and Bioelectronics
  • Neuroengineering and Neurobiology
  • Immunoengineering
• Lower-performing areas include:
  • Biomaterials and Biomolecules
  • Genetics
  • Systems and Synthetic Biology
  • Drug Delivery, Therapy, and Nanomedicine

But the paper is careful: subfield performance can’t be explained solely by the proportion of experimental vs conceptual questions. The authors highlight exceptions, like subfields with high experimental fractions that still perform well, and subfields with more conceptual items but still low accuracy.

That tells us something operational: performance depends on more than experimental-ness. Difficulty, contributor-specific question design, and how well training data covers that subfield likely all play a role.

Literature synthesis (L-Syn): models can draft abstracts, but semantic similarity ≠ factual correctness

BioEVAL’s literature synthesis task is called L-Syn. The model reads abstract-withheld full text (with figure captions retained; figure images excluded) and is prompted to write a publication-style abstract. The generated abstract is scored against the author’s reference abstract using embedding-based semantic similarity.

Across the 218 items, performance varies by subfield dramatically. Reported semantic similarity scores range from 0.09 to 0.72 across subfields.

Some specific results:
- Strongest and most consistent performers include:
- GPT-4o-mini and Gemini-2.5-Flash-Lite, with similarity between 0.63 and 0.72 across all evaluated subfields
- Several other cloud-scale models stay relatively stable with average scores around ~0.61–0.70

Subfield patterns:
- Immunoengineering gets the highest overall similarity
- Bioimaging, Biophotonics, and Optics tend to score lower (clustered around ~0.52–0.63)

But here’s the caveat you should care about

The paper explicitly warns: embedding-based similarity is a measure of semantic agreement, not guaranteed factual fidelity. Sentence embeddings can be insensitive to numerical magnitude and directions of claims, and natural-language-inference scoring may treat missing claims as “neutral” rather than “contradictory.”

So L-Syn scores are best viewed as:
- “How well did the model cover and align with the reference abstract’s semantic content?”
not as:
- “How factually accurate is the model’s draft abstract?”

In practice, this is still useful. The paper supports near-term uses like rapid literature comprehension and drafting summaries—but not yet replacing expert judgment for scientific synthesis.

Multimodal reasoning (MRQs): early evidence models can interpret experimental plots and images

The multimodal benchmark is a pilot set of 10 expert-curated questions. The tasks require interpreting experimental evidence like:
- qPCR amplification curves
- SDS–PAGE gels
- bacterial growth curves
- fed-batch fermentation profiles
- mycoplasma PCR QC gels
- Sanger sequencing chromatograms
- adherent mammalian cell culture phase-contrast images
- enzyme-kinetics plots

What happened in the pilot?

On this 10-question set, several top models hit 8/10 correct, which equals 80% accuracy. The paper lists models achieving this including Gemini-2.5-Flash, Gemini-2.5-Pro, Kimi-K2.5, and Qwen-3.5-9B.

Because the dataset is tiny, the authors stress these are descriptive results rather than definitive rankings.

Still, the key takeaway is: these models can interpret a meaningful fraction of experimental visuals when the task is framed as structured MCQ reasoning.

Explanation alignment behaves differently in multimodal mode

In multimodal MRQs, the paper finds clearer model-dependent variation in explanation-text similarity. Cloud-scale models show more stable explanation alignment, while some edge models (like Qwen-3.5-9B) can score high on answers but show broader variation in how closely their explanations match expert interpretations.

This reinforces a theme from MCQs: answer correctness and explanation alignment don’t always move together.

Model deployment vs capability: edge models can still be useful

One surprisingly practical finding: the paper reports that smaller edge-deployable models can match or outperform larger models on this pilot multimodal set. Again, sample size is small, but it’s a meaningful signal for real-world deployment tradeoffs.

If you’re building a local tool for interpreting experimental readouts, BioEVAL suggests it’s not automatically hopeless—at least for some image modalities and task formats.

Ensemble performance: diversity matters, but correlated errors limit gains

BioEVAL also analyzes how often models share the same correctness patterns and whether majority-vote ensembles improve accuracy.

Key results on MCQs:
- Best single-model accuracy observed: 0.900
- Naive top-3 ensemble: 0.900 (no gain over the best single model)
- Naive top-5 ensemble: 0.911
- Naive top-7 ensemble: (reported as 0.911 for top-5/top-7 in the narrative; best ensemble sizes are summarized separately)
- Exhaustively searched best ensembles (oracle-selected on the same benchmark):
- Best 3-model ensemble: 0.916
- Best 5-model ensemble: 0.925
- Best 7-model ensemble: 0.922

The overall lesson: even when ensembling helps, improvements can be modest because top models may share correlated error patterns. The paper’s correctness-pattern clustering supports this: models often fall into groups where they succeed and fail together.

This matters for anyone tempted to assume “ensemble = big boost.” In bioengineering tasks, the errors may come from shared blind spots—so diversity strategies need to target complementary failure modes, not just add more models.

Direct comparison: where models were strong vs where they struggled

Below is a compact view of the main capability gaps highlighted by BioEVAL.

If you want to understand the full evaluation design and methods, the original BioEVAL paper at https://arxiv.org/abs/2609.30489 is where the study mechanics and scoring details live.

Key Takeaways

Key Takeaways

• BioEVAL is a bioengineering-specific benchmark, built to test not just factual recall but experimental reasoning, literature synthesis, and multimodal evidence interpretation.
• The benchmark contains 608 items: 359 MCQs retained after audit, 218 literature synthesis tasks, and a 10-question multimodal pilot set.
• On MCQs, top cloud-scale models reach 90% accuracy (best reported: Gemini-2.5-Pro), while edge-deployable models are still competitive (GPT-oss-20B at 85%, Qwen-3.5-9B at 80%).
• Models tend to perform worse on experimental MCQs than conceptual MCQs, highlighting a real gap between knowledge and evidence-based reasoning.
• Explanation alignment is informative but imperfect: high answer accuracy doesn’t guarantee strong semantic match to expert rationales.
• Literature synthesis (L-Syn) achieves high semantic similarity in some subfields (up to 0.72), but similarity scores don’t prove numerical or factual correctness.
• The multimodal pilot shows promising early capability (80% on 8/10 for top models), but the dataset is tiny—so results are directional, not definitive.
• Ensembling yields modest gains because top models often share correlated error patterns; diversity strategies need to be more thoughtful than “pick the highest scorers.”
• BioEVAL is designed to be extensible, with standardized evaluation protocols for ongoing benchmark expansion and model testing.