Deep Label-Wise Attentive Temporal Convolutional Networks Improve Medical Coding

# Deep Label-Wise Attentive Temporal Convolutional Networks Improve Medical Coding
Source: [https://arxiv.org/html/2607.25129](https://arxiv.org/html/2607.25129)
Muhammed Yavuz Nuzumlalı Yale University yavuz\.nuzumlali@yale\.edu &Alexander Fabbri Yale University alexander\.fabbri@yale\.edu

###### Abstract

Medical coding is the task of assigning a set of diagnosis and procedure codes for a hospitalization using recorded notes\. It requires aggregating information from different parts of the text and focus to different sections for each individual code, making it a very difficult problem even for professional human coders\. We model the task as a multi\-label text classification problem\. To overcome the mentioned difficulties, we propose a deep neural model consisting of a multi\-layer temporal convolution network \(TCN\) followed by label\-wise attention\. While multi\-layer TCN helps extract a global document representation with the ability to learn relations over very long sequences, label\-specific attention mechanism allows the model to focus on different aspects of the same document for each individual label\. Our method achieves significantly better F\-1 scores \(9% increase\) compared to the previous state\-of\-the\-art model, with a remarkable increase in recall score \(28% increase\), which we believe is the more important metric for a clinical decision support setting\.

Note on the date of this work\.This work was carried out in 2019 at the LILY lab, Yale University, and the manuscript is posted here essentially as it stood at that time\. The experiments and the baselines it compares against reflect the medical coding literature as of 2019\. It is made available as a record of the work rather than as a current account of the field\.

## 1Introduction

Analyzing free\-text Electronic Health Records \(EHR\) is a highly challenging task that has to be performed regularly by health care providers in clinical settings\. Even though such records include extremely valuable information about the medical history of patients, their practical utilization is minimal because of the lack of standards, subjectivity among physicians and the labor intensive review process\. Medical coding is a representative task which requires professional coders to manually and thoroughly review each individual recorded note for a hospitalization, with the aim of assigning a set of standardized International Classification of Diseases \(ICD\) codes corresponding to the procedures and diagnoses occurred during the hospitalization\.

Apart from the challenges inherent in the complicated nature of clinical notes, such as different writing styles and lack of standardization, medical coding problem presents additional issues\. First, the number of labels assigned to a specific note can be very large\. Second, evidence for each individual label may exist at different locations within the note itself, requiring a person to aggregate global contextual information over distant sections of the notes rather than just using local contextual information\. This is especially hard because of the length of the patient note records \(the median number of words in a discharge summary in the MIMIC\-III datasetJohnsonet al\.\([2016](https://arxiv.org/html/2607.25129#bib.bib17)\)is 1,388\)\.

To tackle the problems described above, we propose a deep neural network architecture combining multi\-layer temporal convolutional networks \(TCN\)Baiet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib6)\)with a label\-wise attention mechanism\. Our model has ability to aggregate information over very long sequences thanks to convolution layers with exponentially increasing dilation factors; these dilations provide better awareness of the global context compared to the local context awareness of standard CNN models\. Additionally, the label\-wise attention mechanism enables the model to learn label specific document representations for patient notes by attending to different sections of the text sequence for each label\. We call our methodLabel\-wiseAttentiveTemporalConvolutionalNetworks \(LATCN\)\.

We evaluate our proposal on the freely accessible MIMIC\-III dataset and compare with various related studies, including the SOTA models\. Our model obtains significantly improved performance scores over the state\-or\-the\-art models, especially in terms of recall, which we believe is a more important metric than precision for a clinical decision support system\.

## 2Method

We formulate the problem as a multi\-label text classification task, where the aim is to assign a set of labels from the overall label space for each text document\. The model consists of a multi\-layer TCN followed by a label\-wise attention mechanism, where we use word2vecMikolovet al\.\([2013](https://arxiv.org/html/2607.25129#bib.bib1)\)embeddings pre\-trained over the corpus of all discharge summaries as input\. The architecture is designed to extract a global context\-aware representation for a given document by effectively passing information through the multi\-layer TCN via exponentially dilated convolution filters\. Using the TCN output as input, the label\-wise attention mechanism extracts per\-label dense attentive representation vectors for the document, which are then used directly to predict class probabilities\. A sketch of the architecture is shown in Figure[1](https://arxiv.org/html/2607.25129#S2.F1)\.

![Refer to caption](https://arxiv.org/html/2607.25129v1/x1.png)Figure 1:High level diagram of the proposed LATCN model\.#### Temporal Convolutional Network

We use TCN, as proposed inBaiet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib6)\), for our base architecture to learn a matrix representation for a document\. In our case, the TCN is composed of multiple residual blocks stacked together with a constant filter sizekkand dilation factor ofdℓ=2ℓd\_\{\\ell\}=2^\{\\ell\}whereℓ\\ellis the index of the residual block\. Each residual block includes 2 1\-dimensional CNN layers with given parameterskkanddd, where normalization, non\-linearity, and regularization are provided by weight normalizationSalimans and Kingma \([2016](https://arxiv.org/html/2607.25129#bib.bib46)\), rectified linear units \(ReLU\)Nair and Hinton \([2010](https://arxiv.org/html/2607.25129#bib.bib43)\), and DropoutSrivastavaet al\.\([2014](https://arxiv.org/html/2607.25129#bib.bib45)\)functions respectively\. This architecture provides a receptive field size of2×\(2ℓ\+1−1\)×k2\\times\(2^\{\\ell\+1\}\-1\)\\times kfor each hidden unit at layerℓ\\ell\.

#### Label\-wise Attention Mechanism

After the TCN network transforms the input document matrixD∈ℝde×ND\\in\\mathbb\{R\}^\{d\_\{e\}\\times N\}\(ded\_\{e\}is the embedding vector dimension,NNis text sequence length\) into the latent document matrixH∈ℝdc​h×NH\\in\\mathbb\{R\}^\{d\_\{ch\}\\times N\}\(dc​hd\_\{ch\}is the number of channels in convolutional layers\), we apply a label\-wise attention mechanism proposed byMullenbachet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib5)\)to get individual document vectors for each label, which is the weighted summation ofHHusing attention weights per label\. More formally, we learn an attention matrix parameterU∈ℝdc​h×LU\\in\\mathbb\{R\}^\{d\_\{ch\}\\times L\}whereLLis the number of unique labels\. We calculate document vectorsvℓv\_\{\\ell\}for each label as follows:

A=\\displaystyle A=H⊤​U,\\displaystyle H^\{\\top\}U,\(1\)αℓ=\\displaystyle\\alpha\_\{\\ell\}=e​x​p​\(αℓ\)∑n=1Ne​x​p​\(αℓ​n\),\\displaystyle\\frac\{exp\(\\alpha\_\{\\ell\}\)\}\{\\sum\_\{n=1\}^\{N\}exp\(\\alpha\_\{\\ell n\}\)\},\(s​o​f​t​m​a​x\)\\displaystyle\(softmax\)\(2\)vℓ=\\displaystyle v\_\{\\ell\}=∑n=1Nαℓ​hn\\displaystyle\\sum\_\{n=1\}^\{N\}\\alpha\_\{\\ell\}h\_\{n\}\(d​o​c​u​m​e​n​t​v​e​c​t​o​r\)\\displaystyle\(document\\ vector\)\(3\)

#### Output Layer

To calculate the label probabilities, we use one final linear output layer with a sigmoid activation function\. We perform forward propagation of inputs in a different way than a classical fully connected layer such that each document vectorvℓv\_\{\\ell\}feeds only the output unit corresponding to labelℓ\\ell\. We can formulate this as follows:

y^ℓ=σ​\(wℓ⊤​vℓ\+bℓ\)\\displaystyle\\hat\{y\}\_\{\\ell\}=\\sigma\(w\_\{\\ell\}^\{\\top\}v\_\{\\ell\}\+b\_\{\\ell\}\)\(4\)

#### Loss Function

We use binary cross\-entropy loss, a standard method for multi\-label classification, which is defined as:

ℒB​C​E​\(X,y\)=−∑ℓ=1Lyℓ​l​o​g​\(y^ℓ\)\\displaystyle\\mathcal\{L\}\_\{BCE\}\(X,y\)=\-\\sum\_\{\\ell=1\}^\{L\}y\_\{\\ell\}log\(\\hat\{y\}\_\{\\ell\}\)\+\(1−yℓ\)​l​o​g​\(1−y^ℓ\)\\displaystyle\+\(1\-y\_\{\\ell\}\)log\(1\-\\hat\{y\}\_\{\\ell\}\)\(5\)

## 3Experiments

### 3\.1Dataset

MIMIC\-III111[https://mimic\.physionet\.org](https://mimic.physionet.org/)Johnsonet al\.\([2016](https://arxiv.org/html/2607.25129#bib.bib17)\)is a freely accessible dataset containing data from intensive care unit \(ICU\) hospitalizations over 10 years\. For each hospitalization, the dataset contains a set of encounter notes recorded by health care providers, along with a set of assigned ICD\-9 diagnosis and procedure codes and other structured information\. Summary statistics can be found in Table[1](https://arxiv.org/html/2607.25129#S3.T1)\.

Table 1:Summary statistics for MIMIC\-III dataset\.Following previous studies, we use only discharge summaries as input because they are the most comprehensive notes taken by a physician and serve as a summary of the entire hospitalization period\. We also truncate texts to have a maximum length of 2,500 tokens\. For the scope of this study, we focus on the 50 most common labels observed in MIMIC\-III dataset, leaving the evaluation on the full label set as future work\.

### 3\.2Evaluation Metrics

Although there are many previous studies on ICD code classification, no standard set of metrics exists\. While AUC\-ROC \(Area Under The Curve \- Receiver Operating Characteristic\) and micro/macro averaged F1 are applied by most prior work, precision atkk\(P@k\) and recall atkk\(R@k\) are reported with inconsistentkkvalues, making results hard to compare and interpret\. While P@k can be helpful to assess the correctness of the assigned values offered by a decision support systemMullenbachet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib5)\), R@k assesses how likely a system can prevent users from performing manual analysis of an entire document set to extract any possible missing values\. In clinical settings, high recall is more valuable than high precision because the cost of searching for any missing code could be much higher than the cost of filtering among a set of codes\.

To provide quantitative comparisons with previous work, we report R@5 and P@5 metrics, together with macro/micro averaged AUC\-ROC, precision, recall, and F1 scores\.

### 3\.3Training

We implement and train our model using the PyTorch222https://pytorch\.orglibrary, with AdamKingma and Ba \([2014](https://arxiv.org/html/2607.25129#bib.bib44)\)as the optimization method\. For hyperparameters, we focused on tuning the number of TCN residual blocksℓ∈\{1,2,4,6\}\\ell\\in\\\{1,2,4,6\\\}, number of channelsdc​h∈\{50,100,200,300,400,500,600\}d\_\{ch\}\\in\\\{50,100,200,300,400,500,600\\\}, and filter sizek∈\{4,10\}k\\in\\\{4,10\\\}using grid search\. We selected best performing combination on development dataset which isℓ=4\\ell=4,dc​h=100d\_\{ch\}=100,k=4k=4\. For the other hyperparameters, we followed the suggestions of the original TCN paperBaiet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib6)\)\. Specifically, we use a dynamic learning rate with thereduce on plateaupolicy from 0\.002 towards 0\.0001, and dropout rate of 0\.4\. We train for 200 epochs\.

## 4Results

The overall performance metric scores for our model and previous studies are presented in Table[2](https://arxiv.org/html/2607.25129#S4.T2)\. In general, our best performing method significantly outperforms the SOTA model DR\-CAML for all metrics but macro/micro precision\. F1 goes up 9% on average\.

We observe the most significant improvement on recall scores\. Our model improves the result of DR\-CAML approximately 28%, which is a highly remarkable improvement\. We hypothesize that the strong improvement on recall results from TCN’s ability to learn better generalized models\. We think that the very large receptive field of the proposed TCN model provides better global contextual information gains over the long text sequences compared to CNN models, which can only learn a very tight local contextual information of at most the size of the filter\. The reduction in precision scores \(approximately 7%\) also results from the same phenomenon; while a more generalized model can provide better coverage of the overall label space, a less generalized model can learn to predict a specific subset of labels very accurately, although having diminished performance over the remaining set of labels\.

#### Discussion

In most clinical settings, we believe recall is a more important metric than precision, mainly because any automated system would only be useful as a decision support tool which will be validated by experts\. This scenario will continue to be the same especially for NLP on clinical texts for the foreseeable future, given the performance gap between SOTA systems and human performance\. Therefore, we suggest that automated systems should prioritize recall over precision\.

Table 2:MIMIC\-III performance results on set of most common 50 labels\.

## 5Related Work

### 5\.1Multi\-Label Text Classification

The task of extreme multi\-label text classification extends multi\-label classification to cases of massive label spaces, presenting problems of sparsity and scalabilityLiuet al\.\([2017](https://arxiv.org/html/2607.25129#bib.bib49)\); Youet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib48)\)\.Liuet al\.\([2017](https://arxiv.org/html/2607.25129#bib.bib49)\)present a family of CNN models to tackle a Wikipedia document classification dataset, whileYouet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib48)\)propose a bi\-directional long short\-term memory network and a multi\-label attention mechanism to capture long distance dependencies and word\-label importance\.

### 5\.2ICD Coding

Medical coding presents similar challenges to extreme multi\-label classification because of the enormous label space \(∼\\sim13K for ICD\-9,∼\\sim68K for ICD\-10\) and label imbalance in existing datasets\.

Approaches to ICD classification have largely focused on convolutional architectures\.Rios and Kavuluru \([2018a](https://arxiv.org/html/2607.25129#bib.bib8)\)combine matching networks with a CNN to improve frequent and infrequent label cases\.Mullenbachet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib5)\), meanwhile, use an attentional CNN \(CAML\)\. It extracts a local context\-aware representation of the document, and attention weights learned per label select the most important segments of the document for a particular label\. They also combine label embeddings pre\-trained using code descriptions \(DR\-CAML\) to improve performance\. They match SOTA and use attention weights to offer interpretable predictions\.Rios and Kavuluru \([2018b](https://arxiv.org/html/2607.25129#bib.bib14)\)extend the above approach using a graph\-based CNN architecture to leverage structured relations among ICD\-9 codes and learn hierarchical structure\.

Other approaches have similarly attempted to make use of the characteristics of the input and label spaces\.Wanget al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib13)\)learn a joint word\-label embedding space to build a compatibility metric between label\-word pairs, whileBaumelet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib9)\)suggests a hierarchical attention model with Bi\-GRU network blocks over sentences and documents\.Xuet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib11)\), on the other hand, build an ensemble model to use unstructured \(patient notes and ICD guidelines\), semi\-structured \(ICD code descriptions\), and structured \(lab results, prescriptions\) data\.

### 5\.3Temporal Convolutional Networks

TCNs have outperformed RNNs on standard sequential benchmark tasksBaiet al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib6)\)\. Recently, this model was adapted to several classification tasks[Schwenket al\.](https://arxiv.org/html/2607.25129#bib.bib47); Jianget al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib51)\); Abreuet al\.\([2019](https://arxiv.org/html/2607.25129#bib.bib52)\)\.[Schwenket al\.](https://arxiv.org/html/2607.25129#bib.bib47)use stacked temporal convolutional blocks to learn very deep networks for text classification\.Abreuet al\.\([2019](https://arxiv.org/html/2607.25129#bib.bib52)\)andJianget al\.\([2018](https://arxiv.org/html/2607.25129#bib.bib51)\)incorporate TCNs with attention for sentiment analysis and few\-shot classification, respectively\.

## 6Conclusion

We addressed the problem of assigning diagnosis and procedure codes for a hospitalization from encounter notes as a multi\-label classification problem\. We proposed a neural network architecture that incorporates temporal convolutional network blocks with label\-wise attention to model longer text sequences\. Our model outperforms the previous SOTA on a standard medical coding dataset and greatly improves recall scores compared to previous methods, a key characteristic in real\-world medical settings\. In future work we will experiment with this model on the full label set, in addition to investigating contextual embeddings and leveraging the hierarchical structure of ICD\-9 codes\.

## References

- J\. Abreu, L\. Fred, D\. Macêdo, and C\. Zanchettin \(2019\)Hierarchical attentional hybrid neural networks for document classification\.CoRRabs/1901\.06610\.Cited by:[§5\.3](https://arxiv.org/html/2607.25129#S5.SS3.p1.1)\.
- S\. Bai, J\. Z\. Kolter, and V\. Koltun \(2018\)An empirical evaluation of generic convolutional and recurrent networks for sequence modeling\.arXiv preprint arXiv:1803\.01271\.Cited by:[§1](https://arxiv.org/html/2607.25129#S1.p3.1),[§2](https://arxiv.org/html/2607.25129#S2.SS0.SSS0.Px1.p1.7),[§3\.3](https://arxiv.org/html/2607.25129#S3.SS3.p1.6),[§5\.3](https://arxiv.org/html/2607.25129#S5.SS3.p1.1)\.
- T\. Baumel, J\. Nassour\-Kassis, R\. Cohen, M\. Elhadad, and N\. Elhadad \(2018\)Multi\-label classification of patient notes: case study on icd code assignment\.InWorkshops at the Thirty\-Second AAAI Conference on Artificial Intelligence,Cited by:[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p3.1)\.
- X\. Jiang, M\. Havaei, G\. Chartrand, H\. Chouaib, T\. Vincent, A\. Jesson, N\. Chapados, and S\. Matwin \(2018\)On the importance of attention in meta\-learning for few\-shot text classification\.CoRRabs/1806\.00852\.Cited by:[§5\.3](https://arxiv.org/html/2607.25129#S5.SS3.p1.1)\.
- A\. E\. Johnson, T\. J\. Pollard, L\. Shen, H\. L\. Li\-wei, M\. Feng, M\. Ghassemi, B\. Moody, P\. Szolovits, L\. A\. Celi, and R\. G\. Mark \(2016\)MIMIC\-iii, a freely accessible critical care database\.Scientific data3,pp\. 160035\.Cited by:[§1](https://arxiv.org/html/2607.25129#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.25129#S3.SS1.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.Cited by:[§3\.3](https://arxiv.org/html/2607.25129#S3.SS3.p1.6)\.
- J\. Liu, W\. Chang, Y\. Wu, and Y\. Yang \(2017\)Deep learning for extreme multi\-label text classification\.InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7\-11, 2017,pp\. 115–124\.Cited by:[§5\.1](https://arxiv.org/html/2607.25129#S5.SS1.p1.1)\.
- T\. Mikolov, I\. Sutskever, K\. Chen, G\. S\. Corrado, and J\. Dean \(2013\)Distributed representations of words and phrases and their compositionality\.InAdvances in Neural Information Processing Systems 26,C\. J\. C\. Burges, L\. Bottou, M\. Welling, Z\. Ghahramani, and K\. Q\. Weinberger \(Eds\.\),pp\. 3111–3119\.External Links:[Link](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)Cited by:[§2](https://arxiv.org/html/2607.25129#S2.p1.1)\.
- J\. Mullenbach, S\. Wiegreffe, J\. Duke, J\. Sun, and J\. Eisenstein \(2018\)Explainable prediction of medical codes from clinical text\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),Vol\.1,pp\. 1101–1111\.Cited by:[§2](https://arxiv.org/html/2607.25129#S2.SS0.SSS0.Px2.p1.9),[§3\.2](https://arxiv.org/html/2607.25129#S3.SS2.p1.3),[Table 2](https://arxiv.org/html/2607.25129#S4.T2.1.1.6.4.1),[Table 2](https://arxiv.org/html/2607.25129#S4.T2.1.1.8.6.1),[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p2.1)\.
- V\. Nair and G\. E\. Hinton \(2010\)Rectified linear units improve restricted boltzmann machines\.InProceedings of the 27th international conference on machine learning \(ICML\-10\),pp\. 807–814\.Cited by:[§2](https://arxiv.org/html/2607.25129#S2.SS0.SSS0.Px1.p1.7)\.
- A\. Rios and R\. Kavuluru \(2018a\)EMR coding with semi\-parametric multi\-head matching networks\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),Vol\.1,pp\. 2081–2091\.Cited by:[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p2.1)\.
- A\. Rios and R\. Kavuluru \(2018b\)Few\-shot and zero\-shot multi\-label learning for structured label spaces\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 3132–3142\.Cited by:[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p2.1)\.
- T\. Salimans and D\. P\. Kingma \(2016\)Weight normalization: a simple reparameterization to accelerate training of deep neural networks\.InAdvances in Neural Information Processing Systems,pp\. 901–909\.Cited by:[§2](https://arxiv.org/html/2607.25129#S2.SS0.SSS0.Px1.p1.7)\.
- \[14\]H\. Schwenk, L\. Barrault, A\. Conneau, and Y\. LeCunVery deep convolutional networks for text classification\.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3\-7, 2017, Volume 1: Long Papers,pp\. 1107–1116\.Cited by:[§5\.3](https://arxiv.org/html/2607.25129#S5.SS3.p1.1)\.
- N\. Srivastava, G\. Hinton, A\. Krizhevsky, I\. Sutskever, and R\. Salakhutdinov \(2014\)Dropout: a simple way to prevent neural networks from overfitting\.The Journal of Machine Learning Research15\(1\),pp\. 1929–1958\.Cited by:[§2](https://arxiv.org/html/2607.25129#S2.SS0.SSS0.Px1.p1.7)\.
- G\. Wang, C\. Li, W\. Wang, Y\. Zhang, D\. Shen, X\. Zhang, R\. Henao, and L\. Carin \(2018\)Joint embedding of words and labels for text classification\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2321–2331\.Cited by:[Table 2](https://arxiv.org/html/2607.25129#S4.T2.1.1.7.5.1),[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p3.1)\.
- K\. Xu, M\. Lam, J\. Pang, X\. Gao, C\. Band, P\. Xie, and E\. Xing \(2018\)Multimodal machine learning for automated icd coding\.arXiv preprint arXiv:1810\.13348\.Cited by:[§5\.2](https://arxiv.org/html/2607.25129#S5.SS2.p3.1)\.
- R\. You, S\. Dai, Z\. Zhang, H\. Mamitsuka, and S\. Zhu \(2018\)AttentionXML: extreme multi\-label text classification with multi\-label attention based recurrent neural networks\.CoRRabs/1811\.01727\.Cited by:[§5\.1](https://arxiv.org/html/2607.25129#S5.SS1.p1.1)\.