Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts

arXiv cs.LG Papers

Summary

This paper presents a learning-based framework that models individual scorer behavior via confusion matrices to derive more robust sleep stage labels from multiple experts, improving accuracy on DOD-H and DOD-O datasets.

arXiv:2608.12446v1 Announce Type: new Abstract: Sleep stage classification is important for the diagnosis and management of sleep disorders, yet most automatic staging studies evaluate models against a single reference hypnogram despite known inter-scorer variability. This study investigates whether multi-scored datasets can be used to construct more reliable reference labels from the collective behavior of multiple experts. We use the publicly available DOD-H and DOD-O datasets. EEG (C3-M2) and chin EMG signals were segmented into 30-s epochs, and 30 features were extracted from each modality, yielding 60 features for EEG+EMG. We propose a learning-based hypnogram (LBH) that models the stage-specific behavior of each scorer using confusion matrices derived from machine-learning models. After column normalization, these matrices estimate the probability of each true sleep stage given each scorer's label; probabilities are aggregated across scorers to assign the final label for each epoch. LBH was evaluated with random forest, support vector machine, and multilayer perceptron classifiers under EEG-only and EEG+EMG settings, and compared with the dataset hypnogram (DH) and best-scorer hypnogram (BSH). LBH consistently improved overall performance. The best results were obtained with random forest and EEG+EMG, reaching 86.07% accuracy, 85.46% precision, and 85.29% F1-score on DOD-H, and 86.04% accuracy, 85.21% precision, and 84.70% F1-score on DOD-O. These findings suggest that personalized scorer modeling can improve reference hypnogram construction without discarding information from individual experts.
Original Article
View Cached Full Text

Cached at: 08/14/26, 09:29 AM

# Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts
Source: [https://arxiv.org/abs/2608.12446](https://arxiv.org/abs/2608.12446)
[View PDF](https://arxiv.org/pdf/2608.12446)

> Abstract:Sleep stage classification is important for the diagnosis and management of sleep disorders, yet most automatic staging studies evaluate models against a single reference hypnogram despite known inter\-scorer variability\. This study investigates whether multi\-scored datasets can be used to construct more reliable reference labels from the collective behavior of multiple experts\. We use the publicly available DOD\-H and DOD\-O datasets\. EEG \(C3\-M2\) and chin EMG signals were segmented into 30\-s epochs, and 30 features were extracted from each modality, yielding 60 features for EEG\+EMG\. We propose a learning\-based hypnogram \(LBH\) that models the stage\-specific behavior of each scorer using confusion matrices derived from machine\-learning models\. After column normalization, these matrices estimate the probability of each true sleep stage given each scorer's label; probabilities are aggregated across scorers to assign the final label for each epoch\. LBH was evaluated with random forest, support vector machine, and multilayer perceptron classifiers under EEG\-only and EEG\+EMG settings, and compared with the dataset hypnogram \(DH\) and best\-scorer hypnogram \(BSH\)\. LBH consistently improved overall performance\. The best results were obtained with random forest and EEG\+EMG, reaching 86\.07% accuracy, 85\.46% precision, and 85\.29% F1\-score on DOD\-H, and 86\.04% accuracy, 85\.21% precision, and 84\.70% F1\-score on DOD\-O\. These findings suggest that personalized scorer modeling can improve reference hypnogram construction without discarding information from individual experts\.

## Submission history

From: AmirHossein Eshghi \[[view email](https://arxiv.org/show-email/74af7fc0/2608.12446)\] **\[v1\]**Wed, 12 Aug 2026 17:28:43 UTC \(1,464 KB\)

Similar Articles

Staging by the Book: Automatic Sleep Stage Classification Using Scoring Rules

arXiv cs.AI

This paper presents a deterministic, rule-based sleep staging method that explicitly implements the American Academy of Sleep Medicine (AASM) scoring rules, providing epoch-level natural language explanations. It achieves 60.5% epoch-level agreement with a majority-vote consensus on 50 polysomnography recordings, offering transparency as a complement to opaque deep learning models.