DeepSWE benchmarks indicate that DeepSeek v4 Pro only passes 8% of tasks
Summary
A discussion about DeepSWE benchmarks showing that DeepSeek v4 Pro passes only 8% of tasks, which is surprisingly low compared to its performance on similar tasks.
Similar Articles
DeepSeek-V4-Flash-0731 now far surpassing the DeepSeek-V4-Pro-Preview in benchmarks
DeepSeek's new V4-Flash-0731 model is now far outperforming the V4-Pro-Preview in benchmarks, marking a significant improvement in the model family.
@pilvar222: We ran DeepSeek v4 Pro 0813 on our cybersecurity benchmark, it outperformed EVERY (!) other model at finding vulnerabil…
DeepSeek v4 Pro 0813 outperforms all other models on a cybersecurity vulnerability-finding benchmark, achieving 87.5% CVE rediscovery at pass@3, though with lower precision and run consistency.
DeepSWE Opus 4.8 results have been released.
The results of DeepSWE Opus 4.8 have been released, showcasing its performance on benchmarks.
Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks?
A user reports that DeepSeek-V4-Flash-0731 is unreliable for non-coding office tasks like summarization and meeting notes, failing at concept extraction and speaker understanding despite strong benchmark scores, while Gemma-4-31B performs better.
How good is DeepSeek-V4 Flash, actually?
An evaluation of the performance and capabilities of DeepSeek-V4 Flash, assessing its real-world effectiveness.