Dr. Zhangshu (Joshua) Jiang

LLM Evaluation Researcher & Medical Doctor

About

LLM evaluation researcher and systems builder, and medical doctor working on post-training, benchmark design, and retrieval-augmented systems for high-stakes domains. PhD research examines long-horizon model behaviour over longitudinal clinical data: QLoRA fine-tuning of open-source models with Unsloth, rubric-based evaluation of factuality and grounding, and multi-step reasoning assessment. Built an open-source natural-language-to-SQL system adopted by five research teams, and currently evaluate frontier model outputs as a Medical Fellow at Scale AI. My clinical background means I notice when an answer sounds right but isn’t.

Work

Scale AI

|

Medical Fellow: Model Evaluation and Rubric Assessment (Part-time)

→

Summary

Evaluating frontier-model medical outputs against expert-developed rubrics to assess factual accuracy, reasoning, and safety.

Highlights

Evaluate frontier-model outputs in medicine against expert-developed rubrics for factual accuracy, reasoning quality, and safety.

Contribute to rubric calibration and failure-mode analysis, mostly around spotting answers that sound complete but leave out something that matters clinically.

Feed recurring error patterns back into the evaluation process to make it more consistent over time.

King’s College London, EPSRC DRIVE-Health CDT

|

PhD Researcher: LLM Evaluation and Applied Generative AI

London, United Kingdom

→

Summary

Researching long-horizon model behavior over longitudinal clinical data, focusing on post-training, benchmark design, and retrieval-augmented systems.

Highlights

Build long-horizon benchmarks for temporal and counterfactual reasoning over longitudinal EHRs to catch plausible but incorrect intermediate reasoning.

Fine-tune Qwen-family models using QLoRA/PEFT with Unsloth across five reasoning task categories, evaluating instruction-following, consistency, and robustness under targeted prompting.

Built an eight-domain evaluation suite covering factual grounding, citation support, and adversarial robustness across 200+ constructed queries; reduced unsupported outputs by ~30% against a prompted baseline.

Built a retrieval-augmented generation pipeline over 500+ clinical documents, cutting unsupported recommendations by ~35% versus baseline prompting. Manuscript under review at Nature Communications.

Cleveland Clinic London

|

Neuroscience Fellow and Digital Health AI Fellow (Part-time)

London, United Kingdom

→

Summary

Leading development of clinical decision tools and dashboard metrics.

Highlights

Lead development of AI-assisted decision tools, focusing on interpretability and failure-mode analysis in clinician-facing workflows.

Translate requirements from 15+ clinical domain experts into data models and evaluation dashboards.

Lancashire and South Cumbria Secure Data Environment

|

Clinical AI Research Fellow and Lead ML Architect

→

Summary

Shipped OMCP, an open-source Model Context Protocol server for controlled natural-language access to structured databases.

Highlights

Shipped OMCP, an open-source Model Context Protocol server for controlled natural-language access to structured databases, with audit logging, role-based access control, and privilege separation.

Built a natural-language-to-SQL pipeline with query rewriting, caching, and batching, delivering sub-second responses across federated data sources spanning four organisations.

Packaged and documented the system for adoption across five research teams, cutting partner onboarding time by ~60%.

Processed 2M+ records using transformer-based NLP for entity extraction and normalisation.

Lancashire Teaching Hospitals NHS Foundation Trust

|

Medical Doctor: ICU, Neurosurgery, Acute Medicine

→

Summary

Delivered clinical care across ICU, neurosurgery, and acute medicine while designing real-time deterioration detection tools.

Highlights

Built and deployed a deterioration-detection alert system across four clinical teams. Cut false-positive rates by ~20% without missing more real events.

Worked with real-time multimodal ICU data, where ground truth is delayed and the wrong call has real consequences.

Education

King’s College London, EPSRC DRIVE-Health CDT

→
MPhil/PhD Health Data Science and Medical AI

Courses

Focus: LLM evaluation, post-training, long-horizon benchmarks, counterfactual reasoning, retrieval-augmented systems

University of Sheffield

→
MBChB Medicine Grade:

Courses

GMC full registration with licence to practise

Languages

English

Native

Chinese

Native

Spanish

Professional

Certificates

Retrieval Augmented Generation (RAG) with LangChain

GMC Full Registration with License to Practice

General Medical Council

Advanced Life Support (ALS)

Resuscitation Council UK

Skills

LLM post-training

Python, PyTorch, Hugging Face Transformers, Unsloth, QLoRA/PEFT, supervised fine-tuning, instruction tuning, vLLM, LangChain.

Evaluation and quality

benchmark design, rubric construction, factuality and citation assessment, hallucination evaluation, adversarial testing, failure-mode analysis.

Search and retrieval

retrieval-augmented generation, natural-language-to-SQL, advanced SQL, ETL pipelines, knowledge graphs, Neo4j.

Infrastructure

Docker, FastAPI, REST APIs, GitHub Actions, Azure/AWS/GCP.

Projects

Federated Analytics Platform

Summary

Privacy-preserving analytics across three NHS trusts and 500K patient records, cutting cross-trust query time by 50%.

OMCP (Open-source Model Context Protocol Server)

Summary

Lead author and maintainer of an open-source Model Context Protocol server for auditable LLM access to structured databases via natural-language-to-SQL.

Dyad Labs

Summary

AI-native simulator for evaluating human–AI performance in clinical workflows, comparing clinician-only, model-only, and clinician-plus-model outcomes.

62M-node Biomedical Knowledge Graph

Summary

Retrieval system over 12 ontologies.