annotation

This paper tests whether same-named emotion labels (e.g., fear, surprise) mean the same thing across GoEmotions, XED, and BRIGHTER datasets, finding large scale mismatches, reduced cross-corpus transfer performance, and cases where classifiers give opposite conclusions depending on training resource.

This paper evaluates how well LLMs can replace human annotators for machine translation quality assessment across MQM and ESA schemes, 70 language pairs, and WMT23/WMT25 data, finding that both humans and LLMs remain unreliable in many settings and proposing human-LLM collaborative annotation pipelines.

Goldsmith, an agentic annotation pipeline presented in an arXiv paper, converts small expert-annotated gold sets into structured annotation definitions optimized via an executable loss-guided loop, outperforming direct rewriting, OPRO, APE, and PromptBreeder.

Researchers introduce a theory-grounded benchmark for automatically detecting mental health stigma in online news and social media, evaluating LLMs and toxicity/sentiment classifiers. Results show neighboring constructs poorly capture stigma and that LLMs overpredict stigma unless given explicit operational rules.

This paper demonstrates that large language models (LLMs) achieve annotation quality equivalent to human experts, suggesting the main challenge is minimizing ambiguity in coding rules rather than choosing between human and machine coders.

Introduces ParsHate, a manually annotated dataset of 10,000 Persian tweets for hate speech and target detection, showing moderate performance of current models and emphasizing the need for more advanced methods.

The paper reports human annotation of Quran recitation transcripts to distinguish mistakes from repetitions and repairs, with an evaluator scoring labels and positions, and preliminary evaluations of coding agents showing high performance.

A prototype plugin for Herdr has been released, enabling annotation tasks with keyboard shortcuts, which may replace the need for Plannotator in similar workflows.

Omasnap has been updated with over 70 PRs, adding features like multiline text, cut out sections, and improved redactions, while maintaining its lightweight 800kb binary for Wayland and Hyprland environments.

LightwheelAI and Hugging Face have released EgoSuite-Open100K, a large-scale open dataset of 100,000 hours of egocentric human activities, fully annotated and available in LeRobot format for AI research and training.

This paper introduces MUSE, a full-text, multi-domain scientific knowledge base of Problem-Solution-Rationale triplets mined from research papers, along with an extraction pipeline and initial experiments on rationale-supervised LLM fine-tuning.

Omasnap is a new native Wayland screenshot and annotation tool for Omarchy and Hyprland, offering vector layers, OCR capture, and clean window capture via Wayland protocols.

ScreenMark is a menu bar app for macOS that provides live annotations and a whiteboard overlay for screen drawing.

Obsidian PDF++ is an Obsidian plugin that transforms PDF backlinks into highlight annotations, enabling markdown-based PDF annotation and quality-of-life improvements to the built-in PDF viewer.

This arXiv paper introduces CohortHijack, a robustness audit that removes non-target cells from single-cell query cohorts to test how annotation tools can be manipulated without altering the target cell's expression profile. It shows that structured removal and search strategies can change refined labels in popular pipelines while preserving the target, identifying query cohort composition as a vulnerability surface.

Annotate is a product that uses screen recordings as prompts, launched on Product Hunt.

This paper presents MMLongBench-Doc-V2, a corrected and semantics-aware revision of the MMLongBench-Doc long-document QA benchmark, fixing annotation errors and replacing string matching with an LLM judge, along with a decision procedure for empty-set keys.

This paper argues that the loanword-vs-switch annotation boundary, rather than the choice of model, drives Kazakh-Russian code-switching identification. The authors present a document-level gold LID dataset with an explicit annotation rule and show that naive heuristics and off-the-shelf LID tools fail to distinguish integrated borrowings from genuine code-switches.

The paper presents a methodology for building a large-scale Russian dataset from social media texts to detect presuicidal and anti-suicidal signals, including annotation guidelines and baseline classification experiments.

Notate is a tool that allows users to annotate anything for humans and their agents.