Generative AI Isn’t Ready to Replace Stats Experts—But It Can Help

Generative AI Isn’t Ready to Replace Stats Experts—But It Can Help

Introduction: What happens when GenAI tries to do statistical research?

Generative AI (GenAI) has gone from “cool chatbot” to “possible research assistant” pretty fast. The big question is whether these tools can do real work in statistical research—especially tasks that require both careful reading and correct methodological reasoning. A new paper from the original paper takes a close look at this by testing GenAI on two research-heavy workflows: building a literature review and generating statistical code.

This literature review + code-generation case study focuses on a specific statistical topic: dynamic treatment regime (DTR) estimation using the dynamic weighted ordinary least squares approach—often called dWOLS. The researchers essentially asked, “Can GenAI find the right papers, summarize them without getting lost, identify gaps, and then turn methods from papers into working code?”

To do the evaluation, they used ChatGPT-5 and ScholarAI (a Sept–Nov 2025 release), and they compared GenAI’s output against benchmarks created by domain experts—namely an existing literature review and expert-developed code for dWOLS. Their overall message is blunt but useful: GenAI can speed up parts of the workflow, but it can’t reliably replace methodological expertise.

Why This Matters: The timing is perfect for pilots—but risky for handoffs

This research is significant right now because GenAI tools are already being pulled into academic work workflows—often casually at first (“just draft the summary,” “just generate the code skeleton,” “just find some citations”). But statistics research punishes small mistakes. In literature reviews, wrong citations or missing context can quietly distort what a field believes. In code, “almost correct” implementations can produce results that look plausible while actually using the wrong equations or weights.

A concrete scenario you could run today:
- A PhD student starts a DTR methods review and asks GenAI to “find and summarize dWOLS extensions.”
- Then they use GenAI’s generated R code to implement the method for a simulation study or a real dataset.
- They might notice errors early (syntax issues), but they may not notice subtle differences like incorrect estimating equations or swapped parameter interpretations—especially if they don’t already know what the correct output should look like.

The paper also builds on earlier AI research (like studies reporting hallucinated citations in earlier GPT versions and code-generation bugs in assistants such as Copilot). What feels different here is the focus on statistical methodology, not just generic summarization or broad coding tasks. The case study shows the uncomfortable middle ground: GenAI can sometimes generate correct structure fast, but correctness can break in ways that require expert judgment to detect.

Main Content Sections

1) The experiment design: how the researchers stress-tested GenAI in statistical work

The authors didn’t just ask GenAI random questions. They ran a structured “workflow test” that mirrors how researchers actually work:
1. Identify relevant sources (papers that use/extend dWOLS for DTR estimation, including real-data applications)
2. Synthesize sources (make summaries and extract the parts useful for a review)
3. Identify research gaps (what’s missing, what’s worth studying next)
4. Write a literature review draft
5. Generate R code implementing the methodology from papers, including simulations and package extension

For literature review benchmarking, the authors used the Nadi review as a reference point: a dWOLS-focused literature review completed by domain experts. Their goal was to see whether GenAI could reproduce what experts already found—and ideally find additional relevant work. For code, they tested GenAI on implementing dWOLS and an extension called G-dWOLS (for continuous treatments/doses), and they compared GenAI output to expert code and expected simulation behavior.

Which tools did they test?

They compared:
- ChatGPT using GPT-5 (with thinking mode toggled as needed)
- ScholarAI (Sept–Nov 2025 release)
- ScholarAI GPT (a version accessed via ChatGPT)

They used premium tiers primarily to reduce limits on response volume and PDF uploads. Also: for fairness, they ran identical prompts across models (zero-shot style—no examples embedded in the prompt).

One important limitation they acknowledge: they can’t perfectly prevent these models from drawing on information available on the web (including the pre-existing review itself). They tried to mitigate this by instructing GenAI to ignore the Nadi review and monitoring response traces when possible—but hidden bias can’t be fully ruled out.

2) Literature review workflow: GenAI can find papers, but relevance decisions and citations still wobble

Paper retrieval: GenAI found lots of relevant work—but not everything, and not consistently

Across the three GenAI systems they used, they identified 40 unique papers total. Domain experts then judged:
- 26 relevant
- 14 not relevant

By model:
- ChatGPT (GPT-5) found 26 unique papers, with 22/26 relevant
- ScholarAI found 21 unique papers, with 12/21 relevant
- ScholarAI GPT found 20 unique papers, with 14/20 relevant

They also reported an important miss: only one paper that appeared in the Nadi review was not found by any model. Their hypothesis was keyword mismatch—for example, if a paper discussed dWOLS strongly but didn’t use the same telltale keywords like dWOLS, GenAI could overlook it.

Retrieval “success” isn’t just about coverage

Even when models didn’t fabricate entire papers (a major improvement over older GPT behavior), they still produced problems:
- incorrect author attribution
- broken links (especially in ScholarAI GPT)
- recommending papers that were clearly irrelevant once experts checked

So yes—GenAI can help you discover candidates quickly. But the workflow still requires manual validation before you treat anything as “review-ready.”

Relevance classification: false positives and false negatives show up

When the authors tried to have GenAI categorize papers as relevant vs not relevant, they quantified error rates (based on expert truth labels). For the ChatGPT workflow:
- False positive rate: ~20%
- False negative rate: ~15–22%

In other words, GenAI often sounded confident but still misclassified meaningful fractions of papers. One especially telling detail: some papers were explicitly connected to dWOLS (e.g., with dWOLS in the title/abstract), yet GenAI still sometimes failed to classify them correctly.

That matters because a literature review isn’t just a list—it’s the foundation for how the field frames “what’s known” and “what’s missing.” A 15–20% misclassification rate isn’t catastrophic if you’re doing expert screening, but it’s dangerous if you’re hoping GenAI can do the whole job unattended.

Summary generation: structure is sometimes better than depth

After retrieval, the authors asked GenAI to summarize each paper. They compared:
- summary quality and organization
- apparent hallucination rate
- whether key details got omitted

Main findings:
- Summaries had minimal apparent hallucinations—no fictional papers appeared in summaries.
- But the depth and contextual understanding weren’t reliable.

Key differences:
- ScholarAI was very consistent: it repeatedly used the same subsections in summaries.
- ChatGPT varied more in structure unless prompted very specifically, but sometimes included more methodological detail.

However, both models had issues:
- ScholarAI more often omitted critical details, especially in full-text summarization.
- ChatGPT sometimes included formulas/equations copied from the paper without clear variable definitions, which is basically useless if you’re not already fluent in the notation.

Also, PDF handling was a recurring bottleneck:
- PDFs sometimes weren’t read properly.
- In some cases, ScholarAI produced vague summaries even when it struggled to extract content—without clearly warning the user.

Research gaps: better when context is given, still often repetitive or out-of-scope

Identifying research gaps is where GenAI struggles most, because gaps require interpretation and domain context, not just “what does this paper say?”

They tried three setups:
1. Ask gaps right after summarizing (within the same session)
2. Ask gaps after supplying a list of citations
3. Ask gaps after uploading full PDFs

Result pattern:
- Scenario 1 (no strong context) → gaps were often just repetitions of points already present in the provided material.
- Scenarios 2 and 3 → improved relevance, but many gaps were still:
- already covered by existing literature
- out-of-scope
- vague or based on misunderstanding the method’s purpose (e.g., suggesting things that don’t align with dWOLS’s causal framework)

And although ChatGPT tended to be more “actionable,” both models often needed a domain expert to decide what was real and what was generic brainstorming.

Writing a full review: drafts happen, but correctness issues cluster in citations

When asked to produce a full literature review, ScholarAI couldn’t get there, even after many prompting attempts. The authors then focused on ChatGPT trying both:
- GPT-5 Thinking
- GPT-5 Deep Research

They found:
- Both generated long drafts with structure and flow.
- But both had major citation issues (wrong papers credited to the wrong claims, missing citations, or citations that didn’t match the described method/topic).
- Deep Research wrote better prose, but it struggled with PDF-based constraints because it’s geared toward web search rather than reliably mining user-uploaded documents.

A subtle but crucial point: incorrect citations might not be obviously wrong to someone new to the area. That’s exactly how bad literature reviews get built—confident writing on top of shaky sourcing.

3) Code generation for dWOLS and G-dWOLS: fast code isn’t the same as correct methodology

Now for the part where “almost right” can become quietly dangerous: implementing statistical methodology.

The authors evaluated ChatGPT (GPT-5) for:
1. reproducing simulation studies from the dWOLS paper and the G-dWOLS extension paper
2. analyzing simulated data using dWOLS
3. extending an existing R package DTRreg to support G-dWOLS

Did the code run?

Often, yes—after multiple prompts.

They reported:
- The first code blocks were usually not executable due to small errors (syntax issues, undefined variables).
- Fixes typically required 1–2 additional prompts.
- In a “few situations,” they couldn’t get executable code even after conversational debugging.

So “compile success” is not the real safety bar. The real bar is whether the code matches the paper’s estimating equations and assumptions.

Methodology correctness: small differences showed up as slightly wrong estimates—and sometimes wrong equations

This is the bigger concern. The authors noted cases where:
- ChatGPT omitted parameters expected to be estimated (until corrected)
- parameter estimates were close to expected values, but differences came from incorrect methodology, including:
- incorrect estimating equations
- incorrect weight functions
- sometimes parameter labeling got swapped (estimates for stage 1 vs stage 2 mislabeled)

Even worse: ChatGPT sometimes “justified” its deviations as preserving double robustness—yet those justifications were not supported by the source material. In other words, the model could confidently invent a rationale for why it had changed the math.

That’s a key lesson for statistical programming: if you don’t already know the correct equations, the model can produce code that looks consistent and runs, while still being wrong in a way that affects results.

When ChatGPT didn’t have the source paper PDF

If the dWOLS paper wasn’t uploaded, ChatGPT often tried to use the DTRreg package or wrote its own function.

In those cases:
- it sometimes hallucinated functions that don’t exist in the package
- after correction, it could sometimes use DTRreg correctly by relying on internet-accessed package info

But when it wrote dWOLS “from scratch,” it repeatedly reproduced the same types of errors seen earlier: wrong estimating equations and wrong weights—especially defaulting to the simplest model structure unless the user explicitly provided details.

Extending DTRreg to support G-dWOLS: integration failure was common

The authors attempted to extend DTRreg by prompting ChatGPT to modify the package based on the G-dWOLS paper.

They found that:
- Without package-specific context, ChatGPT created a standalone function—but that wasn’t a correct integration into DTRreg’s structure.
- Even when the authors supplied package source files, the suggested edits were inconsistent with actual code layout.
- Git-based patch instructions and manual edit suggestions didn’t result in a working G-dWOLS implementation.
- The resulting output still failed methodological checks (e.g., wrong number of parameter estimates).

So: GenAI can help you draft code, but robust software engineering for statistical packages—where interfaces, parameterization, and model assumptions must all match—is still mostly a human job.

4) What these results imply for your workflow (and what to do with the risk)

The authors’ overall conclusion is pretty clear:
- GenAI is not an adequate substitute for methodological expertise.
- It can improve efficiency for tasks that leverage retrieval, summarization, and basic debugging—if supervised.

A good way to interpret this is to split research tasks into two categories:

Tasks where GenAI helps most

• Finding candidate papers (with subsequent expert filtering)
• Drafting high-level summaries with clear prompts
• Extracting “what the paper is about” quickly
• First-pass code scaffolding and minor debugging

Tasks where GenAI is most risky

• Automated “relevance” decisions without double-checking
• Identifying gaps without domain oversight
• Producing full literature review drafts with trustworthy citations
• Implementing or extending statistical methodology unless you can verify equations, weights, and outputs carefully

There’s also an uncomfortable meta-point: GenAI can reduce time spent on routine work—but researchers have to spend time re-validating. So the net efficiency gain depends on how often the model’s output is “plausible but wrong.”

And note the “quiet failure” mode:
- code might run
- estimates might look close
- outputs might be labeled correctly
…but the math could still be off by weight functions or estimating equations. That’s exactly the kind of issue that statistical experts are trained to guard against.

5) A simple comparison: what worked, what didn’t, and where each tool fit best

Here’s a high-level comparison based on the observed outcomes in the study.

Key Takeaways

• GenAI can speed up parts of statistical research, especially paper discovery and first-pass summarization—but it cannot be trusted to do critical judgment steps without expert verification.
• In the literature review task, models produced relevant papers without fabricating entire fake citations, but they still had wrong author attributions and false positives/negatives in relevance classification.
• Summaries tended to capture the “main idea,” yet often lacked true methodological depth and contextual understanding—sometimes because PDF reading was unreliable.
• Research gap identification improved when GenAI had structured context, but still often returned gaps that were vague, repetitive, out-of-scope, or misunderstood the method’s causal framing.
• In code generation, GenAI sometimes produced code that ran, but it could still implement the wrong estimating equations or weight functions, leading to “almost right” results that require careful expert checks.
• Extending an existing stats package like DTRreg to support G-dWOLS was not reliably achievable through conversational prompting alone; integration details are too easy for models to get subtly wrong.
• The practical recommendation from this study: treat GenAI as a secondary tool—use it to accelerate workflow steps, but keep methodological validation and citation accuracy firmly under human control.

If you want, I can also turn this into a “checklist” you can use before trusting GenAI outputs for (1) literature review inclusion and (2) statistical code correctness.