I benchmarked models sized 2B to 35B on hard HTML data extraction
Summary
A benchmark comparing AI models ranging from 2B to 35B parameters on a challenging task of extracting structured data from HTML, evaluating their performance and accuracy.
Similar Articles
A 4b model is now beating 30b ones at web research and the reason is not size
A 4 billion parameter open model from the Apodex family outperforms 30 billion parameter models on web research benchmarks, attributed to careful training data and self-verification techniques rather than raw scale, suggesting a more democratic trajectory for AI capability.
HuggingFace benchmark datasets now let you filter by model size
HuggingFace benchmark datasets now allow filtering by model size, enabling comparisons like 'best model under 32B on swebenchverified'.
@nathanhabib1011: best models < 128B params on SWE-bench_pro... @Alibaba_Qwen 3.6 27b still crazy, closely followed by @ornith_ 35B
Tweet highlighting top-performing AI models under 128B parameters on the SWE-bench_pro benchmark, noting Alibaba Qwen 3.6 27B and ornith 35B as leading contenders.
New bench designed for smaller models: ObviousBench.com
ObviousBench is a new benchmark designed specifically for evaluating smaller AI models.
Benchmarking Large Language Models for Safety Data Extraction
This paper benchmarks four large language models (Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, Llama 3.1-70B) for extracting structured information from Safety Data Sheets, finding that text-based extraction with chain-of-thought prompting yields the highest accuracy (84% by Gemini 1.5 Pro) but no model surpasses the 90% threshold required for reliable industrial deployment.