AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Hugging Face Daily Papers Papers
llm-evaluation anti-money-laundering downsampling importance-sampling benchmark-efficiency error-diagnosis graph-prompts

Summary

AMLworld-Compact提出一种重要性加权降采样方法，用于在保留全部非法交易的同时对良性交易分层抽样，使AML基准评测目标减少271至611倍，同时用Horvitz-Thompson权重获得无偏的混淆矩阵估计，并发布了评测集与审计工具。

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.
Original Article
View Cached Full Text

Cached at: 10/12/26, 02:37 AM

Paper page - AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

Source: https://huggingface.co/papers/2610.13627

Abstract

Evaluatinglargelanguagemodels(LLMs)onAMLworld’ssmallesttestsplitsiscostlybecauseeachofoveramilliontargetsrequiresaseparategraphprompt.Samplingreducesthiscost,butsmallbalancedsubsetschangeunweightedprecisionandF1.WeintroduceAMLworld-Compact,anevaluationsetthatretainseveryillicittesttransactionandsamplesbenigntransactionsfromsupervised-scorestrata.ItsHT-Coresetprotocolreducesevaluationtargetsby271xand611xonHI-SmallandLI-Small.Horvitz-Thompsonweightsgiveunbiasedfull-testconfusion-countestimatesforfixedpredictions;recallisexact,whileprecisionandF1areratioestimates.Inaheld-outstudyusingGraphFeaturePreprocessor(GFP)features,oneboosted-treefamilydefinesstrataforevaluatingtheotherover500drawsperconfiguration.Atthereleasedsizes,F1RMSEis0.00-2.58percentagepointsversus4.44-7.41foruniformsamplingwithHTweights;conservativeintervalsremainbroad.Undertheevaluatedgraphprompts,sevenopen-weightLLMshaveweightedprecisionnearthepredict-all-illicitbaseline.Inanoutcome-stratifiedauditof1,000HI-Smallfew-shotoutputsfromonerunpermodel,lexicalcheckspassfor84.5%ofcorrectand56.2%ofincorrectbinaryverdicts,sothesemarkersdonotestablishcorrectness.Wereleasetheevaluationsets,constructioncode,andaudittoolsathttps://huggingface.co/datasets/typhoon-ai/AMLworldCompactEvalandhttps://github.com/nat-nischw/AMLworld-Compact.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.13627

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.13627 in a model README.md to link it from this page.

Datasets citing this paper1

#### typhoon-ai/AMLworldCompactEval Viewer• Updated27 minutes ago • 6.02k • 597

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.13627 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

[2506.13771] LittleBit: Ultra Low-Bit Quantization via Latent Factorization

Reddit r/LocalLLaMA

LittleBit 提出了一种通过低秩潜在矩阵分解与二值化实现超低比特（低至0.1 BPW）LLM 量化的框架，配合多尺度补偿机制，在 Llama2-7B 上以 0.1 BPW 超越了现有 0.7 BPW 方法，实现约 31 倍内存压缩与最高 11.6 倍推理加速。