Generative AI Meets Medical Imaging: Diffusion, LLMs, and Foundation Models

Generative AI Meets Medical Imaging: Diffusion, LLMs, and Foundation Models

Introduction

If you’ve noticed AI suddenly getting good at making things—not just labeling them—you’re paying attention to the right shift. Over the last couple of years, generative AI has moved from “interesting demo” to “real workflow contender,” and medical image processing is getting pulled into the same tide.

This blog post is based on new research from Masahiro Oda’s paper on arXiv. It walks through how today’s generative systems—especially diffusion models for images and large language models (LLMs) for text—are already being applied to medical support tasks. It also explains why a newer idea, foundation models (models trained broadly on huge datasets, then adapted to many tasks), may reshape how we build medical AI in the future.

Why This Matters

Here’s the part that feels especially relevant right now: medical AI has historically been siloed. One model for one task (say, tumor segmentation in one modality), trained on one dataset, optimized for one deployment setting. That approach doesn’t scale well as hospitals add new devices, new imaging protocols, and new rare-disease cases.

What the paper argues—implicitly and explicitly—is that generative AI and foundation models point toward a more “universal” strategy. Instead of building a brand-new pipeline every time, you train or adapt a broader model once, then reuse it across tasks. You can think of foundation models like Google Maps, while previous task-specific models were more like a single static map for one neighborhood. You still need local directions, but you don’t need to redraw the whole world each time.

A scenario where this could be applied today: imagine a radiology department preparing for a busy day and needing draft radiology report language. LLM-based tools can help generate consistent report text and summaries from structured inputs (e.g., findings detected by image models). Meanwhile, diffusion-based systems can support dataset expansion—creating realistic variations for training when real labeled data is scarce. The paper gives concrete examples like diffusion-generated mammograms used to improve mammogram diagnostic support, and diffusion-based segmentation methods that generate segmentation outputs through a controlled “denoising” process.

And compared to earlier AI progress: the research frames diffusion models and LLMs as a jump from discriminative AI (classify/detect/segment) toward high-information generation (create images, generate text, and—crucially for medicine—generate training-ready signals). Then foundation models add a second jump: not just “generating outputs,” but generating useful capability across tasks with far less per-task retraining burden.

Diffusion Models Turn “Noise” Into Medical Structure

To understand why diffusion models took over image generation, it helps to picture them as a careful restoration process. First, the model takes real data (like an image) and gradually adds noise until the image becomes pure static. Then it learns to reverse the process: starting from noise, it repeatedly removes noise to reconstruct something that matches the target distribution.

The paper explains diffusion models using the classic diffusion probabilistic model idea: diffuse forward (add noise) and denoise backward (learn to remove noise). A commonly used variant is DDPM (Denoising Diffusion Probabilistic Model), where generation happens through many steps—typically around 1,000 steps for high-quality results in the paper’s description.

Why diffusion models beat older image generators (in practice)

Before diffusion became popular, many medical imaging researchers leaned on VAE (variational autoencoders) or GAN (generative adversarial networks).

• VAE training tends to be more stable and easier to evaluate, but the images often lack expressive detail.
• GAN can create strikingly detailed images (the paper points to StyleGAN2 as an example), but training can be unstable and progress harder to monitor.
• Diffusion-based services like DALL·E 3, Stable Diffusion, and Midjourney produce more visually convincing images than those older approaches in many settings—and this is closely tied to diffusion models’ stepwise denoising approach.

The “U-Net + conditions” recipe that makes diffusion controllable

In a typical DDPM implementation for generation, the reverse diffusion step is handled by a model like U-Net. The paper notes that U-Net is an encoder-decoder FCN-style architecture often familiar from medical segmentation work.

For medical use, the key is conditioning: you don’t just want random anatomy-like images, you want images tied to something meaningful—text prompts, label maps, or other signals. The paper describes how text or other conditions get embedded and fed into the model at each denoising step. In short: diffusion becomes controllable when the model repeatedly denoises while “listening” to the condition.

Diffusion in Medicine: From Training Data Boosts to Segmentation Engines

The paper is careful about a crucial point: generated medical images are not real patient data and therefore can’t directly replace clinical evidence for diagnosis or therapy. But they can still be very valuable—especially when your goal is to improve medical AI.

Data augmentation that actually looks like medicine

Traditional augmentation (rotations, flips, color jitter, Mixup, etc.) is useful but limited: it often produces variations that don’t capture real-world clinical diversity (shape variability, lesion appearance, acquisition differences).

Diffusion-based augmentation is different because it can generate realistic, diverse samples that reflect anatomical structure and disease-related patterns—if you control generation properly.

A specific example the paper highlights: a study that used Stable Diffusion to generate mammogram images with embedded tumors, where tumor position and size were controlled through text prompts. When those synthetic images were added to training, the resulting diagnostic support model performed better than training on real images alone.

A big win: diffusion-based segmentation

Segmentation is where diffusion models shine in medical imaging. The paper frames this as: diffusion can output segmentation results as images, by running the denoising process while using extracted embeddings from the input medical images as conditions.

The research cites MedSegDiff as an early representative method. It generates segmentation images through reverse diffusion using medical-image embeddings as conditions. Compared to earlier FCN and ViT-based approaches such as SegNet, nnU-Net, TransUNet, and diffusion variants like MedSegDiff-V2, the paper reports MedSegDiff achieved higher segmentation accuracy.

Diffusion segmentation improvements: architecture and noise design

The paper then points to a pattern in the field: people improve segmentation by improving (1) how information from the input and current denoising state is fused, and/or (2) how noise is modeled.

For example, the paper describes MT U-Net, a U-Net variant for diffusion segmentation. It takes two inputs:
- the output from the previous reverse diffusion step, and
- the original medical image.

It uses cross-attention (so the model can “match” relevant features) and a multi-size transformer module (so it captures information at multiple spatial scales). The reported result: better segmentation than several FCN/ViT approaches and earlier diffusion methods.

Another interesting comparison: noise type.
Most diffusion models use Gaussian noise. But the paper describes work for binary segmentation where using Bernoulli-distributed noise (discrete two-value noise) improved results. In their tests on MRI and CT binary segmentation, the Bernoulli-noise diffusion model outperformed FCN/ViT methods and Gaussian-noise diffusion approaches.

Here’s a compact way to see the “comparison space” the paper explores:

When text and images mix: segmentation with limited labels

Another practical bottleneck in medical imaging is label scarcity—especially with limited annotated datasets.

The paper introduces TextDiff, where segmentation training leverages both image information and text descriptions (when available). It uses a diffusion-based image encoder and a Clinical BioBERT text encoder, then merges them via cross-modal attention.

A concrete datapoint from the paper: TextDiff achieved high performance using only 150 X-ray images for training. That’s a big deal because it suggests multimodal conditioning can reduce the amount of annotated image data you need.

Going beyond 2D: video segmentation with temporal reasoning

Diffusion isn’t limited to still images. The paper describes Diff-VPS for polyp segmentation in colonoscopy videos. Two innovations stand out:
1. Multi-task learning (adding classification and detection subtasks) to provide contextual guidance.
2. A temporal reasoning module trained to estimate the target frame from previous frames, helping the diffusion segmentation incorporate temporal information.

The takeaway: segmentation improves when the model treats video not as separate frames, but as a time-consistent sequence.

LLMs for Medical Text: Radiology Reports, Summaries, and Structured Findings

Generative AI isn’t only about images. Medical workflows also depend heavily on text: radiology report writing, clinical summaries, structured findings, and even anonymization.

The paper reviews how modern text generation systems rely on Transformers, where an attention mechanism helps the model understand how words relate across a sequence. It notes that systems like ChatGPT and Gemini are based on large-scale Transformer models, and that scaling data, model parameters, and compute has been a major driver of quality improvements.

Scaling laws and why LLM training looks nothing like old-school ML

The paper describes Scaling Laws for Neural Language Models, highlighting the consistent theme: more training data, larger models, and more compute tends to increase performance.

It gives a timeline for GPT scale:
- GPT-1: 4.5GB data, 117M parameters, trained on 8 GPUs
- GPT-2: 40GB data, 1.5B parameters
- GPT-3: 570GB data, 175B parameters, trained on 10,000 GPUs
- GPT-3.5: 355B parameters (as described in the paper’s narrative)

The point isn’t just “big is better.” It’s that the training regime requires self-supervised learning, because manual annotation at that scale is impossible.

Medical LLMs: domain-tuned models reduce the “wrong vocabulary” problem

General LLMs may sound fluent, but medicine has domain-specific language and clinical structure. The paper discusses medical-domain LLMs, including BioBERT (trained on PubMed and PMC; 110M parameters) and larger models such as BioMedLM, GatorTronGPT, and PMC-LLaMA.

For Japan-focused language, the paper notes Japanese-tuned LLM releases like ELYZA-japanese-Llama-2-7b and Llama-3-ELYZA-JP-70B, plus models from Preferred Networks and Japan’s National Institute of Informatics (e.g., LLM-jp-3 172B).

Practical medical support examples: report drafting and image-to-text

The paper points to commercial services where LLMs support medical writing:
- Therapixel (France) with MammoScreen (tumor detection + draft radiology reports including findings and impressions)
- RADPAIR (US) offering dictation-to-report generation support tools

It also describes research work like ChatCAD, which detects tumors in images and then uses an LLM to generate explanatory text based on those results.

So the model stack in real use often looks like:
1. image model extracts findings (detection/segmentation),
2. LLM drafts narrative text,
3. humans review/edit for clinical safety and accuracy.

Foundation Models: One Engine, Many Medical Tasks

Now the big conceptual leap: foundation models.

Where earlier AI was usually built task-by-task, foundation models are trained broadly and then adapted. The paper emphasizes traits like:
- trained on large-scale cross-task datasets,
- transferable to many downstream tasks,
- capable of reducing the repeated effort of training from scratch for each task.

It uses SAM (Segment Anything Model) as the image-processing foundation model example and GPT as the language example.

Why foundation models matter for medicine specifically

Medical imaging faces two brutal constraints:
1. collecting huge labeled datasets is expensive,
2. rare diseases have limited data.

Foundation models—and the ability to adapt with small amounts of data—can reduce both issues.

The paper describes ways to adapt foundation models:
- Zero-shot: adapt via prompts without task-specific training data
- Few-shot: fine-tune with a small dataset
- Parameter-efficient learning: adapt large models without retraining everything

The medical foundation model “family”: SAM-style vs medical-domain foundation models

The paper notes that SAM, trained mainly on natural images, can still work for medical images, but often with less accuracy than domain-specific models.

That’s why many medical foundation models exist. The paper gives several concrete examples:

• MedSAM: a segmentation-oriented foundation model trained on over 1.57 million images across many modalities (CT, MR, X-ray, ultrasound, mammography, OCT, endoscopy, dermoscopy, fundus, pathology). It uses bounding boxes as prompts.
• MedSAM-2: extends segmentation to 3D and video by adapting SAM 2 capabilities (treating one spatial axis as temporal).
• BiomedCLIP: image+text multimodal foundation model built on CLIP, using PubMedBERT for text and trained on PMC-15M (about 15 million figure-caption pairs from 4.4 million articles). It supports classification and visual question answering.
• BioViL-T: X-ray images + radiology report text model, trained on about 174,100 image-text pairs from MIMIC-CXR v2, supporting classification and image-to-text generation.
• PathAsst: pathology images + text model for classification and text generation, trained on over 207,000 image-text pairs from sources like PubMed.
• Prov-GigaPath: pathology single-modality foundation model pretrained on 1.3 billion image patches extracted from 171,000 whole-slide images (using methods like DINOv2).
• RETFound: ophthalmology foundation model for fundus and OCT images trained on 1.6 million images using masked autoencoder ideas.

Training foundation models: self-supervised learning (SSL) is the engine

The paper explains the usual recipe:
1. pretrain with SSL on many unlabeled images,
2. fine-tune on labeled images for each downstream task.

Common SSL techniques for vision foundation models include:
- Contrastive Learning (CL) (make augmented views of the same image similar; push apart different images)
- Masked Image Modeling (MIM) (mask image patches and reconstruct)
- hybrids like CL + MIM

It lists representative methods such as SimCLR, MoCo, SimSiam, DINO, and CLIP for multimodal contrastive learning, and Masked Autoencoder variants for reconstruction-based pretraining.

Domain gap is the silent killer

One practical warning the paper makes: even strong foundation models can underperform if the downstream domain is too different from the data used for pretraining. Natural-image models like SAM may struggle in medical domains unless adapted with medical data.

So the “future direction” isn’t only “make bigger models,” but also “make medical-relevant models” when possible.

Building National-Scale Medical Foundation Models: Data, Compute, and Bias Awareness

The paper’s future-focused section gets very concrete. It argues that big tech’s model scale is hard for academic groups to match—but national-scale approaches are possible.

It calls out three scaling drivers:
- more training data
- more model parameters
- more training compute

Then it argues you don’t necessarily need to import everything from scratch—you can leverage national data and national compute.

The missing piece: patient-population fit

There’s also a fairness angle. The paper notes that bias related to country or race matters in healthcare AI, and (in its view) no foundation model tailored specifically to Japanese patient populations has been realized yet. That implies Japan could benefit from foundation models trained on Japanese datasets to improve fit and reduce mismatch.

Large-scale medical image datasets are already forming

The paper gives an example from Japan: a Medical Image Big Data Cloud Platform built by Japan’s National Institute of Informatics (NII), storing over 400 million medical images across radiological, endoscopic, ophthalmologic, and pathology modalities.

It also mentions how labeling large datasets is feasible through human-in-the-loop workflows combined with AI assistance:
- a report describes annotating multi-organ abdominal segmentation on 8,448 CT cases (about 3.2 million 2D slices) in three weeks
- their workflow used multiple segmentation models (Swin UNETR, nnU-Net, U-Net) and prioritized corrections using inter-model disagreement (uncertainty)

Synthetic data and simulation: filling the labeling gap

Another practical lever is synthetic/simulation data. The paper points to the XCAT Phantom, which can generate virtual images with controlled variables (age, gender, body size, organ structure, motion), including realistic noise and artifacts.

It also notes simulation-driven training for retinal vessel segmentation, where training with simulated OCT angiography generated from vascular development simulation improved segmentation accuracy compared to real data alone in their reported results.

This suggests an approach many teams are already considering: sim-to-real training pipelines, where synthetic data teaches the model structure while real data fine-tunes it to reality.

Compute access: supercomputers and subsidized programs

Finally, scaling requires training compute. The paper argues that distributed training on many GPUs is realistic if researchers tap into university/research-institution supercomputers.

It notes that these environments often support Python-based development, and while usage costs exist, subsidy programs can lower the barrier—making large-model training more achievable for non-big-tech teams.

Key Takeaways

• Diffusion models generate images by reversing a noise-addition process; for high quality the paper describes ~1,000 reverse steps, typically using U-Net and conditioning via text/labels.
• In medicine, diffusion models are especially useful for data augmentation and segmentation, not direct clinical diagnosis with synthetic images.
• Diffusion-based segmentation methods (like MedSegDiff, MedSegDiff-V2, and MT U-Net) push accuracy by improving conditioning, architecture fusion, and even the type of noise (Gaussian vs Bernoulli for binary masks).
• LLMs (LLM-based text generation) support medical workflows such as report drafting and structured narration, and domain-tuned models like BioBERT help address medicine-specific language needs.
• Foundation models trained on large cross-task datasets reduce the need to rebuild AI from scratch per task; the paper highlights both general foundation models (SAM) and medical-domain ones (MedSAM, BioMedCLIP, Prov-GigaPath, etc.).
• Building truly effective national medical foundation models is feasible by combining national-scale datasets (the paper cites 400M+ images in Japan’s platform), human-in-the-loop labeling, synthetic/simulation data, and access to large compute—while also addressing population and data bias.

If you want, tell me what part you care about most—diffusion for segmentation, LLMs for radiology text, or foundation-model training—and I’ll tailor an “implementation-minded” walkthrough around that piece.