Tag
The article points out that in the field of large model evaluation, there are positions with high demand, high salary, and scarce talent, especially in systematically identifying model weaknesses, specialized expert systems, and multi-step agent data evaluation. Daily salaries for related talents can reach 30k-80k.
A method using LLMs as a jury to build or validate food metadata, improving data quality and consistency.
This paper examines the validity of using LLMs as data annotators, focusing on Portugal's AMALIA-9B model for European Portuguese. It finds that while the model shows agreement with human coders, its reliance on surface correlates raises concerns about construct validity, with implications for sovereign LLM programs.
The tweet observes that frontier AI labs are spending billions on hiring diverse professionals (poets, musicians, accountants, etc.) to annotate massive datasets, calling it a brute-force bet that seems to be working.
Indian workers, paid 250 rupees per hour, strap phones to their heads to record themselves doing household chores, providing training data for AI robots. They film over 90 different scenes and angles of actions every day, highlighting the labor issues behind AI training.
Scale AI CEO Alexandr Wang shares how Paul Graham's 'Schlep Blindness' essay inspired the company's focus on solving the unglamorous but critical problem of building high-quality data sets for machine learning.
This paper presents the construction of a Korean evaluation-annotated corpus (EVAD) for fine-grained aspect-based sentiment analysis in e-commerce reviews using Semi-Automatic Symbolic Propagation. It evaluates KoBERT and KcBERT models on the dataset, achieving high F1 scores in aspect-value pair recognition.
This paper introduces Annotator Policy Models (APMs) by Apple, which use interpretability techniques to infer annotators' internal safety policies from their labeling behavior without requiring additional annotation effort. The authors demonstrate that APMs can accurately model these policies and distinguish between sources of annotation disagreement, such as operational failures, policy ambiguity, and value pluralism.
This paper presents a large-scale analysis of four harmful language detection datasets, examining how annotator characteristics and linguistic features interact to influence annotation variation. It highlights intersectional effects and warns against generalizing findings across different datasets.
Tendem by Toloka is a platform that connects AI developers with human experts for data annotation and training.