@LiorOnAI: Fireworks just launched the Specialized Intelligence Index. It benchmarks models on real work across healthcare, legal,…

Fireworks just launched the Specialized Intelligence Index.

It benchmarks models on real work across healthcare, legal, cybersecurity, finance, customer support, productivity, and software.

General benchmarks tell you how a model performs on standardized tasks.

SII is about whether a model can actually do a specific job well enough to be useful in practice.

That means testing things like legal work, clinical tasks, customer support investigations, finance workflows, and security reviews against practitioner-defined standards.

Here’s an example: depthfirst’s dfbench v1.

On security, dfbench v1 scores three defensive jobs:

1. Finding vulnerabilities
2. Validating that the findings are real,
3. Checking whether the audit still holds after the code changes.

The benchmark includes 253 real-software examples and 910 vulnerabilities.

depthfirst then post-trained its dfs-large1 model, based on GLM 5.2, with Fireworks using reinforcement learning.

They credit the gains to three things:

1. penalizing wasted effort
2. penalizing too many findings
3. training detection and validation together

On their production workload, depthfirst says dfs-large1 set a new Pareto frontier.

Fireworks runs the evals, partners approve scores before publication, and each benchmark stays as its own result instead of getting collapsed into one overall ranking.

Define the real job. Measure models on that job. Train against the failures. Run the eval again.

Fireworks (@FireworksAI_HQ): Today we’re launching the Specialized Intelligence Index (SII): one destination for real-work benchmarks across industries, built by the teams that use them every day.

Hear from Fireworks co-founder @the_bunny_chen on the importance of specialized benchmarks: