Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
Summary
This paper presents a method to optimize GPT-5.2 temperature for Naranjo causality assessment in pharmacovigilance, achieving significant agreement improvements via Bayesian hyperparameter optimization with a novel composite metric (EWACS).
View Cached Full Text
Cached at: 07/07/26, 04:38 AM
# Optimizing Large Language Models for Causality Assessment in Pharmacovigilance: Developing a Performance Metric as Objective for Bayesian Hyperparameter Optimization
Source: [https://arxiv.org/abs/2607.03704](https://arxiv.org/abs/2607.03704)
[View PDF](https://arxiv.org/pdf/2607.03704)
> Abstract:Background: Growing individual case safety report \(ICSR\) volumes have intensified demand for scalable automated causality assessment\. Large Language Models \(LLMs\) show promise, yet performance on clinically demanding tasks remains suboptimal and inference\-time hyperparameter optimization has not been investigated\. Objective: To develop a Gaussian Process \(GP\)\-compatible optimization objective and investigate whether temperature optimization improves LLM\-expert agreement on Naranjo causality assessment of FAERS ICSRs\. Methods: Expert causality assessments were performed on 723 stratified FAERS cases\. OpenAI's GPT\-5\.2 was evaluated using chain\-of\-thought \(CoT\) prompting\. Four composite metrics were developed: Weighted Cosine Similarity \(WCS\), Information\-Weighted Agreement Score \(IWAS\), Entropy\-Weighted Agreement and Cosine Similarity Score \(EWACS\), and Consensus\-Weighted Cosine Similarity \(CWCS\) and Bayesian optimization using a GP surrogate with Probability of Improvement \(PoI\) acquisition was applied across temperature \[0, 2\]\. Results: GPT\-5\.2 outperformed prior biomedical LLMs at baseline \(T = 0\), achieving 74\.1% agreement on question 5 and 65\.4% on question 10 of Naranjo algorithm\. Entropy analysis identified these as the sole informative optimization targets\. Temperature showed no systematic population\-level effect \(\\b\{eta\} = 0\.002, p = 0\.959\)\. EWACS\-guided Bayesian optimization improved causality classification agreement from 45\.0% to 72\.0% \(\+27 pp\), with the largest gain in Doubtful cases \(\+42\.9 pp\)\. Conclusion: EWACS was identified as the optimal GP\-compatible metric\. The absence of a universal temperature optimum indicates LLM performance is driven primarily by ICSR content, yet case\-specific temperature selection produced meaningful improvements, supporting temperature optimization for LLM\-assisted pharmacovigilance\.
## Submission history
From: Maurizio Sessa Dr\. \[[view email](https://arxiv.org/show-email/fc202b9a/2607.03704)\] **\[v1\]**Sat, 4 Jul 2026 04:49:14 UTC \(7,705 KB\)Similar Articles
Medical Causal Hypothesis Verification with Large Language Models
This paper presents a preliminary study evaluating the accuracy of large language models in verifying causal medical hypotheses, finding that while they exhibit strong recall, they often fail to provide valid scientific evidence or reject unsupported claims.
The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance
This paper systematically evaluates the impact of classification model selection within the InferBERT framework for causal adverse drug event detection, finding that domain-specific pre-training (BioBERT) outperforms both simpler models and larger LLMs like Med-LLaMA.
Analysis of Prompt Engineering for Drug Toxicity Prediction
This paper investigates the impact of prompt engineering on drug toxicity prediction using large language models, finding that natural variance in LLM outputs outweighs prompt fine-tuning, while chemoinformatic feature extraction improves model performance.
Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment
The paper benchmarks five large language models on multi-sensor physical hazard assessment, revealing that all tested models fail to produce precautionary warnings when multiple sensors are simultaneously elevated below their individual safety limits, while achieving near-perfect accuracy on single-sensor violations.
Leveraging Large Language Models for Systematic Literature Review of Disease Spread Models
This paper develops an LLM pipeline for automating systematic literature reviews in disease spread modeling, comparing the performance of GPT-4.1 and GPT-5.0 against human-conducted reviews.