@AntCaveClub: What exactly is Harness? Harness = Evaluation Harness. In AI, "harness" is industry jargon – a set of tools to "harness" a model and run standardized evaluations. The industry standard is EleutherAI's lm-e…

What Exactly Is a Harness?

Harness = Evaluation Harness.

In AI, “harness” is industry jargon — a tool used to “strap in” a model and run standardized evaluations.

The industry standard is EleutherAI’s lm-evaluation-harness (GitHub 14K+ stars). Almost all open-source models use it for MMLU, GSM8K, and HumanEval.

DeepSeek Harness = DeepSeek’s self-built evaluation framework.

There are three roles: Researcher (designs evaluation methodology), Engineer (builds systems), Product Manager (external productization) — this is a full team structure from methodology to engineering to product, not a side project run by interns.

Is It Important?

Extremely important. This might be the most undervalued position in DeepSeek’s hiring posts.

Reasons:

1. Without an evaluation framework, an AI lab is blind.

You train a new model. How do you know it’s better than the previous version?

• “I feel like it answers better” → useless.
• “HumanEval score went from 82% to 85%” → that’s a usable conclusion.

Without your own harness, you can only use public benchmarks from others. The problem:

2. Public benchmarks are already contaminated.

Public benchmarks like MMLU, GSM8K, HumanEval — every model optimizes for them. It’s like every student memorizing the same exam answers: scoring 100 doesn’t mean you’re smart; it means you memorized well.

DeepSeek needs its own private benchmarks, private evaluation sets, and private scoring system. That’s the mission of the Harness team.

3. Evaluation is the core of the training loop.

Train model → run evaluation → analyze weaknesses → modify training strategy → retrain → re-evaluate.

How fast this loop runs depends on how strong your harness is. If your harness can automatically run 100 private evaluation sets and produce results in 10 minutes, your iteration speed crushes teams that are still manually running MMLU.

4. They are hiring now, which shows they are accelerating.

“Interviewing daily + posting ads everywhere” → this team is being built from scratch. It means DeepSeek judges that their current evaluation capabilities are no longer sufficient.

This implies: DeepSeek may be preparing a larger model that requires a more complex evaluation system.

Do I have one? Does Hermes have one?

Hermes has one, but not at DeepSeek’s level.

Hermes has an evaluating-llms-harness skill (exists in the skill directory, but maybe not installed locally), using EleutherAI’s lm-evaluation-harness — the public, open-source, community-maintained version.

Differences:

Maintainer

• EleutherAI lm-eval-harness: Open-source community
• DeepSeek Harness: DeepSeek’s own team

Evaluation sets

• EleutherAI lm-eval-harness: Public (MMLU/GSM8K, etc.)
• DeepSeek Harness: Likely includes private evaluation sets

Customizability

• EleutherAI lm-eval-harness: Moderate, community plugin mechanism
• DeepSeek Harness: Fully customizable, can evaluate however you want

Real-time updates

• EleutherAI lm-eval-harness: Community update pace
• DeepSeek Harness: Driven by internal needs, iterates faster

Strategic value

• EleutherAI lm-eval-harness: A tool
• DeepSeek Harness: Part of core competitiveness

Hermes uses the lm-eval-harness level — sufficient for individual developers or small teams, but not on the same scale as DeepSeek’s self-built system.

Didn’t DeepSeek have one before?

Probably not — they likely had “people who could run evaluations” but not a “dedicated team + productization.”

In the early stages of many AI labs:

• Researcher A wrote a Python script to run MMLU.
• Researcher B forked lm-eval-harness and tweaked it.
• Researcher C wrote their own set.
• Result: everyone’s evaluation results didn’t match, and no one knew which version of the model had actually improved.

When a lab grows to DeepSeek’s scale (serious competition at OpenAI/Anthropic level), a dedicated Harness team becomes necessary.

So this hiring signal means: DeepSeek is upgrading from “lab outputs” to “industrialized evaluation system.”

Who uses it?

All serious AI labs either use or build their own harness:

• OpenAI: What: Self-built. Status: Earliest to build own evaluation system, SimpleEvals, etc.
• Anthropic: What: Self-built. Status: Has internal private benchmark sets.
• Google DeepMind: What: Self-built. Status: BIG-Bench came from them.
• Meta (FAIR): What: lm-eval-harness + self-built. Status: Mix of open-source and self-built.
• Mistral: What: lm-eval-harness + self-built. Status: Same as above.
• DeepSeek: What: Building Harness team. Status: Shifting from hybrid to fully self-built.
• Individuals/Small Teams: What: lm-eval-harness. Status: Sufficient.

What It Means for You

If you want to apply to DeepSeek, these three roles are worth serious consideration:

• Harness Researcher → Suitable for academic background, designing evaluation methodology (how to measure accurately, avoid data leakage, design private benchmarks).
• Harness Engineer → Suitable for engineering background, building distributed evaluation systems, automated pipelines, visualization dashboards.
• Harness Product Manager → Suitable for someone who understands AI and product, turning the evaluation system into a product that is easy to use both internally and externally.

Key info:

• Final interview is with him (the hiring manager) → high interview weight.
• One written test + three interviews → standard DeepSeek process; the written test is tough.
• Researcher: both intern and full-time → lower entry barrier, proves urgent need.

In a word: Harness is the “measuring instrument” of an AI lab. Without it, you’re building a rocket with your eyes closed. DeepSeek is seriously building this team, which means they’re not building a small rocket.

Tianyi Cui (@tianyi): As a newly established department, the DeepSeek Harness team has ambitious goals and heavy workloads, and we are still very understaffed. I interview every day and post ads everywhere… There are three roles:

Harness Researcher (intern/full-time): https://t.co/7oV3DVuPfH Harness Engineer (full-time/intern): https://t.co/b9HjmV3J8I Harness