Instruction Tuned Large Language Models for Assisting Brain Surgery Through Procedural Alignment and Decision Support Using PPO Reinforcement Learning

Abstract

The integration of large language models (LLMs) into clinical decision-making remains a critical challenge, especially in high-risk domains such as neurosurgery. This study presents a novel framework that leverages instruction-tuned LLMs optimized using Proximal Policy Optimization (PPO) reinforcement learning to assist brain surgery through procedural alignment and decision support. We begin by fine-tuning a transformer-based LLM on domain-specific surgical protocols and neurosurgical dialogue datasets using supervised instruction tuning. To further enhance procedural adherence and mitigate hallucinations, we introduce a reward model guided by expert-annotated signals such as factual accuracy, stepwise protocol fidelity, and relevance to surgical context. PPO is employed to iteratively refine the model's responses through a feedback loop, optimizing both language coherence and domain-specific reliability. Experimental evaluations on simulated neurosurgical benchmarks demonstrate that our model outperforms both instruction-tuned and PPO-only baselines in terms of procedural accuracy and decision support relevance. The results indicate that reinforcement learning with human feedback, when tailored to surgical requirements, significantly improves trustworthiness and alignment in LLM outputs. This research contributes a critical step toward the deployment of explainable, reliable AI assistants for neurosurgical procedures.