AI4Doctor: An EMR Centric Clinical LLM

Overview

This paper introduces AI4Doctor, a 32 billion parameter Chinese language clinical LLM. It is built on the premise that existing medical LLMs (Med-PaLM 2, GPT-4V Medical, HuatuoGPT and similar) over-rely on literature and exam style training, and under-use the richer, patient specific reasoning signal locked inside electronic medical records (EMRs).

The authors' fix is an EMR centred training pipeline applied to more than three million de-identified, multi-centre inpatient records, spanning 46 departments at the PLA General Hospital.

The Training Pipeline

The pipeline has four stages.

1. Continued pretraining

Continued pretraining on the de-identified EMR corpus plus medical literature and textbooks, using a novel "Special Temporal Grid" representation. This encodes multi-resolution timing, trends and missingness across a patient's hospitalisation (admission through discharge), rather than relying on fixed interval resampling.

2. Curriculum based instruction fine-tuning

Organised via a "task cube" crossing NLP ability, specialty and clinical application (discharge summaries, admission records, treatment plans, and so on).

3. Clinical alignment

Alignment via Odds Ratio Preference Optimisation (ORPO), a lightweight reinforcement style method that folds in doctor expert preferences and EMR derived signals (diagnostic priors, risk thresholds) at roughly one tenth the compute cost of full RLHF.

4. Hybrid retrieval and inference layer

Pulls similar EMR cases and guideline snippets at query time, across major clinical domains.

Results

Complex clinical questions (senior physician scoring)

AI4Doctor led by a 6.5 point margin over the next best system, and scored notably ahead of the senior physicians' own average performance.

Medical documentation generation

Average of 67.73 across diagnosis plans, admission and postoperative records, and discharge summaries.

Public benchmarks

This beat every other tested model bar GPT-4. Notably, the smaller 13B variant outperformed larger 34B rivals, which the authors read as evidence that EMR grounded training beats raw parameter scaling.

Ablation style gains

• Retrieval augmentation added a further 10.4% lift in consistency ("hit" rate) and 6.0% in fluency.
• ORPO alignment improved all eight measured clinical quality dimensions (comprehensiveness, risk and side effect awareness, professionalism, accuracy, and so on) by 3 to 10%.

CMB-Clin benchmark

CMB-Clin mixes physician, nurse, pharmacist, technician and graduate exam question sets. AI4Doctor's 55.70 average trailed only GPT-4 (59.46) and beat every other tested model, general or medical, open or closed source, including HuatuoGPT and ChatGLM2 6B by a wide margin.

New Benchmark: 301MedQA

Alongside the model, the authors release 301MedQA, a new benchmark of 2,277 EMR grounded clinical questions:

It is built from the same multi-centre EMR repository and intended as a closed book test of episode grounded reasoning rather than exam style recall.