Is this chart lying to me? Automating the detection of misleading visualizations

# Is this chart lying to me? Automating the detection of misleading visualizations

Source: https://arxiv.org/html/2508.21675

Jonathan Tonglet¹,²,³, Jan Zimny¹,², Tinne Tuytelaars², Iryna Gurevych¹

¹Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE
²Department of Electrical Engineering, KU Leuven
³Department of Computer Science, KU Leuven

www.ukp.tu-darmstadt.de

## Abstract

Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccurate conclusions. Prior work has shown that both humans and multimodal large language models (MLLMs) are frequently deceived by such visualizations. Automatically detecting misleading visualizations and identifying the specific design rules they violate could help protect readers and reduce the spread of misinformation. However, the training and evaluation of AI models has been limited by the absence of large, diverse, and openly available datasets. In this work, we introduce Misviz, a benchmark of 2,604 real-world visualizations annotated with 12 types of misleaders. To support model training, we also create Misviz-synth, a synthetic dataset of 57,665 visualizations generated using Matplotlib and based on real-world data tables. We perform a comprehensive evaluation on both datasets using state-of-the-art MLLMs, rule-based systems, and image-axis classifiers. Our results reveal that the task remains highly challenging. We release Misviz, Misviz-synth, and the accompanying code.

github.com/UKPLab/acl2026-misviz

## 1 Introduction

Misleading visualizations are charts that distort the underlying data, typically by violating design principles, leading readers to draw inaccurate conclusions. While many arise from unintentional design errors, misleading visualizations are also deliberately crafted by malicious actors to spread disinformation and manipulate public understanding, especially during crises such as the COVID-19 pandemic, where misleading charts circulated widely on social media.

Prior work has shown that both humans and MLLMs are easily deceived by such visualizations in question-answering tasks. The deceptive features in these charts, or *misleaders*, often reside in subtle details easily missed by readers, such as axis tick intervals. Furthermore, misleaders are highly diverse: the latest taxonomies identify over 70 distinct types spanning a wide range of chart types, including bar charts, pie charts, and choropleth maps. In some cases, multiple misleaders affect the same visualization.

Figure 1 shows 12 real-world examples of misleading visualizations. Appendix A explains how these visualizations misrepresent their underlying data table.

**Figure 1:** Examples of the 12 types of misleaders included in Misviz.

Automatically classifying whether a visualization is misleading and identifying which misleaders affect it, if any, can enable timely warnings to chart designers and readers and help prevent the spread of misinformation. This task is framed as a multi-label classification problem. While early work relied on rule-based systems called linters, recent studies have explored the use of MLLMs. However, these approaches were evaluated on distinct datasets, which are either small or closed-access, limiting comparability and hindering progress.

In this work, we introduce Misviz, a large, diverse, and open benchmark comprising 2,604 real-world visualizations spanning 12 types of misleaders. It reflects scenarios in which detection models could flag visualizations published on the web. In Misviz, 70% of the visualizations contain up to three misleaders, while the remaining 30% are non-misleading. To support model training, we also release Misviz-synth, a synthetic dataset generated with Matplotlib using real-world data tables. Misviz-synth reflects scenarios in which detection models assist chart designers in identifying misleaders unintentionally introduced into their charts. The dataset includes not only the visualizations but also their underlying data tables, Python code snippets, and axis metadata, enabling the training of chart de-rendering models.

We conduct extensive experiments with three approaches: (a) state-of-the-art MLLMs, (b) a new rule-based linter that inspects axis metadata for design rule violations, and (c) new classifiers that take the visualization alone or in combination with the axis metadata as input. For (b) and (c), we fine-tune DePlot to extract axis metadata as an intermediate step. Our experiments address the following research questions (RQs).

**RQ1:** Which type of model performs best on real-world or synthetic instances?

**RQ2:** Can detection models trained on synthetic instances generalize to real-world cases?

**RQ3:** Can axis extraction models trained on synthetic instances generalize to real-world cases?

Our results show that MLLMs perform best on real-world visualizations, while linters and image-axis classifiers outperform them on synthetic ones, benefiting from the availability of training data for both axis extraction and misleader detection. While the fine-tuned DePlot can extract axes from Misviz-synth, it does not generalize well to Misviz, reducing the performance of the linter and classifier.

In summary, our contributions are as follows:

(1) We introduce Misviz and Misviz-synth, the first large-scale open datasets for misleading visualization detection.

(2) We propose a new linter and a new classification method that combines image and extracted axis metadata as input.

(3) We conduct a comprehensive evaluation and error analysis, highlighting the strengths and weaknesses of each method and identifying directions for future work.

## 2 Related Work

| Dataset | Instances | Misleader types | Chart types | % non-misleading | Open access | Real-world | Multi-label | Axes, table, code |
|---------|-----------|-----------------|-------------|------------------|------------|-----------|------------|-------------------|
| MISCHA-QA (Arif et al., 2024) | 8,201 | 43 | 49 | ✓ | ✗ | ✗ | ✗ |
| DCDM (Maciborski et al., 2025) | 24,480 | 53 | 51 | ✓ | ✗ | ✗ | ✗ |
| Alexander et al. (2024)-design misleaders | 1,460 | 7 | > 5 | 50 | ✗ | ✓ | ✗ | ✗ |
| Lo and Qu (2025) | 150 | 21 | > 5 | 16 | ✓ | ✓ | ✗ | ✗ |
| Misvisfix (Das and Mueller, 2026) | 450 | 74 | > 5 | 20 | ✓ | ✓ | ✓ | ✗ |
| Misviz-synth (ours) | 57,665 | 12 | 5 | 39 | ✓ | ✗ | ✗ | ✓ |
| Misviz (ours) | 2,604 | 12 | > 5 | 31 | ✓ | ✓ | ✓ | ✗ |

**Table 1:** Existing datasets for misleading visualization detection.

The first attempts to detect misleading visualizations relied on rule-based systems called linters. These linters assume the availability of the underlying data table or the chart code, which restricts their applicability to real-world scenarios.

Fan et al. and Biselli et al. overcome these limitations by extracting tables using OCR tools from real-world visualizations before applying rule checks. However, the accuracy of these real-world linters depends heavily on the quality of the intermediate OCR step, which can vary widely. Real-world linters have previously been evaluated in small-scale user studies with human-in-the-loop correction of OCR errors.

Others have explored the potential of MLLMs for the task. Lo and Qu evaluated four MLLMs with different prompts on a dataset of 150 real-world visualizations, sourced from the corpus of Lo et al. for misleading cases. They found that detection accuracy decreased as more misleader types were included in the prompt.

Alexander et al. focused on GPT-4, using visualizations from the social media platform X. However, access to this dataset requires a paid API, and reproducibility is further hindered by the platform's frequent removal of posts. In parallel work, Das and Mueller proposed a prompt with which state-of-the-art MLLMs achieve high accuracy on a subset of the corpus of Lo et al. Recently, Maciborski et al. fine-tuned a convolutional neural network for the task, achieving high accuracy on synthetic instances. Table 1 compares prior datasets with Misviz and Misviz-synth.

Misviz is over fifteen times larger than the dataset of Lo and Qu. Unlike Alexander et al., it does not rely on paid APIs for data collection and ensures long-term access to all instances by archiving them on the Wayback Machine. Misviz-synth is two to seven times larger than other synthetic datasets, and includes several more misleaders and chart types. In contrast to other synthetic datasets, Misviz-synth provides the underlying table, code, and axis metadata. The latter is necessary to fine-tune DePlot for axis extraction and answer our research questions.

## 3 Misviz

### 3.1 Selected Misleaders

Misviz covers 12 types of misleaders, selected from the 74 categories defined in the taxonomy of Lo et al., based on four key criteria.

First, we excluded misleaders that are rarely observed in real-world scenarios. To determine this, we used misleader frequency statistics from the corpus of Lo et al. and discarded all categories with fewer than 15 instances. Second, we removed reasoning misleaders, i.e., misleaders that do not directly break chart design rules and are deceiving only in the context of a specific claim. Third, we remove misleaders which confuse rather than deceive. As noted by Lo et al., the taxonomy includes both misleaders that distort the underlying data, the focus of this work, and others that may hinder readability or clarity without altering the interpretation of the data, such as *missing titles* or *overplotting*. Fourth, we excluded misleaders that require specific domain knowledge to be identified. For example, using red to represent Democrats and blue to represent Republicans in a chart violates color conventions, but detecting this misleader requires familiarity with U.S. politics. Such misleaders require domain expertise that is beyond the reach of crowdworkers.

We define each selected misleader briefly below. They cover together 62.3% of all instances from the real-world corpus of Lo et al. Each of them is represented with an example in Figure 1 and in Appendix A.

**Misrepresentation:** the value labels displayed do not match the sizes of their visual encodings; e.g., bars may be drawn disproportionately to their corresponding numerical values.

**3D:** the visualization includes 3D effects, distorting the size of visual encodings.

**Truncated axis:** an axis does not start from zero, thus exaggerating differences between values.

**Inappropriate use of pie chart:** a pie chart does not display data in a part-to-whole relationship.

**Inconsistent binning size:** a variable, such as years or ages, is grouped in unevenly sized bins.

**Discretized continuous variable:** a continuous variable is cut into discrete categories, thus exaggerating the difference between boundary cases.

**Inconsistent tick intervals:** the ticks in one axis are evenly spaced, but their values are not, e.g., the tick values sequence is 10, 20, 40, 45.

**Dual axis:** there are two independent and parallel numerical axes with different scales.

**Inappropriate use of line chart:** a line chart is used in unusual ways, e.g., with categorical data.

**Inappropriate item order:** the tick labels of an axis are sorted in an unconventional way, e.g., dates are not shown chronologically.

**Inverted axis:** an axis is displayed in a direction opposite to conventions.

**Inappropriate axis range:** the axis range is either too narrow or too wide relative to the data distribution.