Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
Summary
This paper presents a method to optimize GPT-5.2 temperature for Naranjo causality assessment in pharmacovigilance, achieving significant agreement improvements via Bayesian hyperparameter optimization with a novel composite metric (EWACS).
View Cached Full Text
Cached at: 07/07/26, 04:38 AM
# Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
Source: [https://arxiv.org/abs/2607.03704](https://arxiv.org/abs/2607.03704)
[View PDF](https://arxiv.org/pdf/2607.03704)
> Abstract:Background: Growing individual case safety report \(ICSR\) volumes have intensified demand for scalable automated causality assessment\. Large Language Models \(LLMs\) show promise, yet performance on clinically demanding tasks remains suboptimal and inference\-time hyperparameter optimization has not been investigated\. Objective: To develop a Gaussian Process \(GP\)\-compatible optimization objective and investigate whether temperature optimization improves LLM\-expert agreement on Naranjo causality assessment of FAERS ICSRs\. Methods: Expert causality assessments were performed on 723 stratified FAERS cases\. OpenAI's GPT\-5\.2 was evaluated using chain\-of\-thought \(CoT\) prompting\. Four composite metrics were developed: Weighted Cosine Similarity \(WCS\), Information\-Weighted Agreement Score \(IWAS\), Entropy\-Weighted Agreement and Cosine Similarity Score \(EWACS\), and Consensus\-Weighted Cosine Similarity \(CWCS\) and Bayesian optimization using a GP surrogate with Probability of Improvement \(PoI\) acquisition was applied across temperature \[0, 2\]\. Results: GPT\-5\.2 outperformed prior biomedical LLMs at baseline \(T = 0\), achieving 74\.1% agreement on question 5 and 65\.4% on question 10 of Naranjo algorithm\. Entropy analysis identified these as the sole informative optimization targets\. Temperature showed no systematic population\-level effect \(\\b\{eta\} = 0\.002, p = 0\.959\)\. EWACS\-guided Bayesian optimization improved causality classification agreement from 45\.0% to 72\.0% \(\+27 pp\), with the largest gain in Doubtful cases \(\+42\.9 pp\)\. Conclusion: EWACS was identified as the optimal GP\-compatible metric\. The absence of a universal temperature optimum indicates LLM performance is driven primarily by ICSR content, yet case\-specific temperature selection produced meaningful improvements, supporting temperature optimization for LLM\-assisted pharmacovigilance\.
## Submission history
From: Maurizio Sessa Dr\. \[[view email](https://arxiv.org/show-email/fc202b9a/2607.03704)\] **\[v1\]**Sat, 4 Jul 2026 04:49:14 UTC \(7,705 KB\)Similar Articles
The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance
This paper systematically evaluates the impact of classification model selection within the InferBERT framework for causal adverse drug event detection, finding that domain-specific pre-training (BioBERT) outperforms both simpler models and larger LLMs like Med-LLaMA.
Can Large Language Models Imitate Human Speech for Clinical Assessment? LLM-Driven Data Augmentation for Cognitive Score Prediction
This paper proposes a large language model-driven data augmentation framework using GPT-5 to generate synthetic oral monologues from written anchors for cognitive score prediction from speech. A similarity-guided selection strategy consistently reduces prediction error, particularly for minority low-score participants.
PSK@EEUCA 2026: Fine-Tuning Large Language Models with Synthetic Data Augmentation for Multi-Class Toxicity Detection in Gaming Chat
This paper presents a system for the EEUCA 2026 shared task on toxicity detection in gaming chat, achieving 4th place by fine-tuning Llama 3.1 8B with synthetic data augmentation. It highlights a 'validation trap' phenomenon where high validation scores do not correlate with test performance due to dataset distribution shifts.
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models
Independent study shows 227M-parameter hypernetwork adds zero gain over well-crafted few-shot prompts for tool-use in 3B Llama, achieving 79.7% of GPT-5 performance at 10× lower latency.
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Researchers introduce MedSP1000, a 1,638-case interactive benchmark derived from standardized patient scenarios to evaluate LLMs as dynamic clinical agents across multi-turn encounters. Results show even the best model (GPT-5.5) completes only 60.4% of expert rubric items, suggesting current LLMs are not yet reliable enough for clinical practice.