evaluations

Tag

Cards List
#evaluations

@BenjaminDEKR: This is quite impressive, and solidifies Google as a world class AI lab not to be underestimated.

X AI KOLs Following ↗ · yesterday Cached

Gemini 4 Neon debuts at #2 on the Vals Index with 70.9%, just 0.4 points behind Claude Fable 5.2 at less than half the cost per task, while taking #1 on the Finance and Legal indexes.

0 favorites 0 likes
#evaluations

@OpenAI: @ApolloResearch will assess whether our models could covertly work against their developers’ intentions—and whether our…

X AI KOLs ↗ · 4d ago Cached

OpenAI announced that Apollo Research will assess whether its models could covertly act against developers' intentions and whether safeguards can detect such scheming behavior across training, safety testing, and internal use.

0 favorites 0 likes
#evaluations

GPT-Rosalind-6 System Card Addendum (6 minute read)

TLDR AI ↗ · 2026-11-02 Cached

OpenAI发布 GPT-Rosalind-6 系统卡增补,说明该模型复用 GPT-6 Astra 权重,并在 Genomics 与生物化学能力/安全评测中报告了相较于 GPT-Rosalind-5.5 的显著提升及相应的防护措施。

0 favorites 0 likes
#evaluations

All the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models

arXiv cs.AI ↗ · 2026-10-27 Cached

该论文提出“多证据整合差距”概念,发现视觉语言模型(如Gemma-3-27B)能正确恢复所有局部事实,但在需要联合约束满足的任务上全局有效率极低(仅19.75%),并通过隐藏态探针和激活替换实验揭示了跨层深度的可干预性。

0 favorites 0 likes
#evaluations

Is anyone concerned about agent reliability or spend??

Reddit r/AI_Agents ↗ · 2026-10-17

A tweet from an SF-based co-founder questioning why companies seem unconcerned about rising AI agent spend and lack robust eval suites or training environments when modifying agent harnesses, asking if others observe similar patterns.

0 favorites 0 likes
#evaluations

Four years of AI progress across seven capabilities, adjusted for changes in the benchmark

Reddit r/ArtificialInteligence ↗ · 2026-10-04

An analysis of four years of AI progress across seven capability areas, with results adjusted for changes and drift in the underlying benchmarks to better reflect true model improvement.

0 favorites 0 likes
#evaluations

@DanKornas: AI workflow stacks get messy fast. This repo keeps the pieces together. Output is an open-source TypeScript framework f…

X AI KOLs Timeline ↗ · 2026-10-03 Cached

Output is an open-source TypeScript framework for building AI workflows and agents aimed at Claude Code users, keeping prompts, evaluations, tracing, cost tracking, orchestration, and credentials unified in one codebase. Key features include workflow folders, version-controlled prompts, automatic tracing, LLM-as-judge evaluations, and durable orchestration powered by Temporal, released under the Apache 2.0 license.

0 favorites 0 likes
#evaluations

Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

arXiv cs.CL ↗ · 2026-09-30 Cached

This paper introduces Fable, a controlled dataset of 190,911 English prompt variants derived from 174K real user prompts, and evaluates 34 open-weight LLMs to show that models mirror higher-level rhetorical and lexical qualities of non-native speakers' prompts, producing lower-quality and less-fluent responses for less fluent users.

0 favorites 0 likes
#evaluations

AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

AutoDataBench introduces a controlled, data-centric testbed to isolate and evaluate LLM agents' "data intelligence" — their ability to diagnose, organize, and construct training data — while holding non-data factors fixed, showing trajectories can also be reused for mid-training gains.

0 favorites 0 likes
#evaluations

Unofficial Jev plugin for coding agents: best practices, an API reference, and 150+ community projects. Evals included.

Reddit r/artificial ↗ · 2026-09-27

An unofficial Jev plugin for coding agents has been released, providing best practices, an API reference, and links to over 150 community projects, with evaluations demonstrating a 96% pass rate in coding tasks compared to other plugins.

0 favorites 0 likes
#evaluations

@undefinedKi: Palantir's AI platform runs inside some of the most secure organisations on earth. Their architecture docs show what an…

X AI KOLs Timeline ↗ · 2026-09-26 Cached

Palantir's AI platform architecture for secure organizations highlights ontology-based tools, model agnosticism, and rigorous logging and evaluations for agent systems.

0 favorites 0 likes
#evaluations

@omarsar0: Own your intelligence stack, folks. You can't scale a company to the frontier by renting intelligence. Custom models, h…

X AI KOLs Timeline ↗ · 2026-09-22 Cached

The tweet emphasizes the need for companies to own their AI intelligence stack, including custom models, harnesses, and evaluations, rather than renting, with a quote from Gabe Pereyra discussing challenges at Harvey.

0 favorites 0 likes
#evaluations

Priorities and principles for effective third party assessments

OpenAI Blog ↗ · 2026-09-22 Cached

OpenAI outlines priorities and principles for effective third party assessments to enhance AI safety, emphasizing independent scrutiny, shared standards, and deep access for rigorous evaluation.

0 favorites 0 likes
#evaluations

We need to talk about Irregular

Reddit r/singularity ↗ · 2026-09-20

The article reveals that misconfigurations at Irregular, an AI security company connected to Effective Altruism, led to AI agents accidentally accessing real systems during cybersecurity evaluations, raising concerns about AI safety testing practices.

0 favorites 0 likes
#evaluations

@Saboo_Shubham_: JUST SHIPPED: 8 free Agent skills with evals. 100% free and open-source. Here are my top 3 picks: /advisor-orchestrator…

X AI KOLs Timeline ↗ · 2026-09-13 Cached

The article announces the release of eight free, open-source AI agent skills with evaluations, highlighting top picks like an orchestrator using multiple AI models, a reader simulator, and a project analyzer.

0 favorites 0 likes
#evaluations

@trySynara: SWE-2 is now available in Synara through your Devin subscription. Cognition’s most capable coding model yet matches rec…

X AI KOLs Following ↗ · 2026-09-10 Cached

Cognition has introduced SWE-2, a coding model that matches recent frontier models on leading evaluations while reducing costs by up to 70%, now accessible through Synara with Devin subscriptions.

0 favorites 0 likes
#evaluations

@mayaislam_ai: AI ≠ ChatGPT. The real stack is: Models + RAG + Agents + Memory + Security + Evals. The moat? How they work together.

X AI KOLs Timeline ↗ · 2026-08-29 Cached

A tweet highlighting that AI involves more than just ChatGPT, emphasizing the importance of integrating models, RAG, agents, memory, security, and evaluations.

0 favorites 0 likes
#evaluations

Ant launches Ling-3.0-flash-Fin for finance workflows; OpenRouter access is free for one month

Reddit r/ArtificialInteligence ↗ · 2026-08-28

Ant's Ling team has released Ling-3.0-flash-Fin, a finance-enhanced AI model designed for financial workflows, with free access on OpenRouter for one month and plans to open-source the weights.

0 favorites 0 likes
#evaluations

@Saboo_Shubham_: We have got some really interesting skills with proper evals here: https://github.com/Shubhamsaboo/awesome-llm-apps/tre…

X AI KOLs Following ↗ · 2026-08-23 Cached

A GitHub repository featuring over 100 open-source AI agent apps and skills with proper evaluations, compatible with multiple LLMs including Claude, Gemini, GPT, and open-source models.

0 favorites 0 likes
#evaluations

Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture

arXiv cs.AI ↗ · 2026-08-18 Cached

This paper critiques existing evaluations of AI moral reasoning for focusing on moral values while overlooking moral norms, and proposes a research agenda to develop standardized methods and datasets for assessing normative reasoning in large language models.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback