Prompt engineering is a powerful tool to optimise the use of generative AI in clinical settings, not only in psychiatry. While large language models (LLM) have shown potential across a range of tasks - from data extraction to psychotherapy - their clinical utility hinges on how prompts are designed. Current reporting is often limited to non-clinical applications or vignettes. A structured approach to prompt engineering could enhance LLM performance. Here, we introduce “clickBrick,” a systematic method to test and refine prompts for extracting psychopathological features from psychiatric records based on our previous work, and evaluate their downstream impact on diagnostic prediction. We applied increasingly structured prompts to locally-run LLMs for extracting expert-annotated symptoms from admission notes. Classifiers trained on these outputs aimed to predict discharge diagnoses (ICD-10) in a dataset of 2,520 in-patient cases. Extraction accuracy varied significantly with prompt design (balanced accuracy: 71.45% to 93.59% for best prompts; 50% to 69.82% for worst). Chain-of-Thought prompts outperformed others in 7/12 symptoms, though simpler prompts worked best for suicidality. Diagnostic classifiers trained on best-prompt outputs also showed superior performance for several diagnoses. In additional experiments, psychiatric emergencies and use of coercive measures were accurately predicted up to 79% of the time. Prompt engineering - particularly through iterative expert-led design - is critical for unlocking LLMs’ clinical potential In mental health research and beyond, it offers a scalable path to improve AI interpretability, accuracy, and real-world relevance.
Title: Prompt engineering for mental health: unlocking the clinical potential of large language models
Speaker: Gerrik Verhees, Germany
Breaking Hot Topics webinar series: Following a record number of submissions to the Breaking Hot Topics Symposium at the 38th ECNP Congress in Amsterdam, the ECNP is proud to launch the Breaking Hot Topics webinar series. This new online series features selected runner-up presentations that showcase the most current and compelling science in neuropsychopharmacology.
The series aims to extend the conversation beyond the Congress, foster visibility for emerging work, and connect researchers across disciplines and borders.
