@onloglogn: Introducing RSI-Inference Bench from Apple ML Research http://rsi-inference.com https://github.com/apple-aiml-research/…
Summary
Apple ML Research introduces RSI-Inference Bench, a benchmark and environments built from real-world vLLM and SGLang inference-engineering tasks to test whether AI agents can improve the systems that run AI, reporting a best-agent pass@1 of just 26.7% after spending $100K in API tokens and 50K GPU hours.
View Cached Full Text
Cached at: 10/17/26, 12:57 PM
Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 http://rsi-inference.com 💻 https://github.com/apple-aiml-research/ml-rsi-inference-bench…
Can AI agents improve the systems that run AI?
We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering:
🚀 Day-0 Model Support ⚡ Inference Optimization 🔧 Bug Fix
→ Unsaturated: best agent at 26.7% pass@1 → Comprehensive coverage of the inference stack → Verifiable rewards: binary and continuous
Watch the video to find out more!
GitHub & BitBucket HTML Preview
Source: https://htmlpreview.github.io/?https://github.com/apple-aiml-research/ml-rsi-inference-bench/blob/main/docs/leaderboard.html Enter URL of the HTML file to preview:
or prepend to the URL:http://htmlpreview\.github\.io/?https://github\.com/twbs/bootstrap/blob/gh\-pages/2\.3\.2/index\.html
Similar Articles
InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
InferenceBench is a benchmark that evaluates AI agents on optimizing LLM inference speed using an H100 GPU across multiple bottleneck scenarios. Results show agents improve over naive baselines but frequently converge on single frameworks and underperform simple hyperparameter searches, indicating a need for better exploration strategies.
@Lyubh22: Coding benchmarks are saturating. AI4Research is the next frontier. Thrilled to see our MLS-Bench (https://mls-bench.co…
Announcing MLS-Bench, the first AI4Research benchmark to gain broad community adoption, testing AI agents on 140 executable tasks across 12 domains to propose modular ML improvements. The post includes leaderboard scores for models like Claude Opus 4.6 and GPT-5.4.
AA-AgentPerf-Local: Benchmarking local AI agents on laptops and workstations
Artificial Analysis open-sources AA-AgentPerf-Local, an inference benchmarking tool that replays real agent trajectories to measure how fast agentic AI runs on laptops and workstations, with initial results for NVIDIA DGX Spark, RTX 5090, AMD Ryzen AI Halo, and MacBook Pro M5 Pro.
GreenBench: Benchmarking Energy Efficiency and Carbon Footprint of Open-Source LLM Inference on Apple Silicon
GreenBench is a benchmarking framework that evaluates the energy efficiency and carbon footprint of open-source LLM inference on Apple Silicon, revealing a 30-40x per-token energy advantage over datacenter GPUs and identifying optimal models for different use cases.
@EinsiaAI: 1/ Recursive self-improvement (RSI) depends on agents improving how AI systems are trained —not just tuning hyperparame…
The article presents AI4AI-Bench, a benchmark evaluating AI agents' ability to improve training algorithms, showing low performance scores and high exploration costs across ten research repositories.