Arc Sentry outperformed LLM Guard 92% vs 70% detection on a head to head benchmark. Here is how it works.

Reddit r/artificial Tools

Summary

Arc Sentry is a new pre-generation prompt-injection detector that reads a model’s internal residual stream, achieving 92% detection with 0% false positives versus LLM Guard’s 70%/3.3% on a 130-prompt benchmark.

I built Arc Sentry, a pre-generation prompt injection detector for open-weight LLMs. Instead of scanning text for patterns after the fact, it reads the model’s internal residual stream before generate() is called and blocks requests that destabilize the model’s information geometry. Head to head benchmark on a 130-prompt SaaS deployment dataset: Arc Sentry: 92% detection, 0% false positives LLM Guard: 70% detection, 3.3% false positives The difference is architectural. LLM Guard classifies input text. Arc Sentry measures whether the model itself is being pushed into an unstable regime. Those are different problems and the geometry catches attacks that text classifiers miss. It also catches Crescendo multi-turn manipulation attacks that look innocent one turn at a time. LLM Guard caught 0 of 8 in that test. Install: pip install arc-sentry GitHub: https://github.com/9hannahnine-jpg/arc-sentry If you are self-hosting Mistral, Llama, or Qwen and want to try it, let me know.
Original Article

Similar Articles

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

arXiv cs.LG

This paper presents an evaluation methodology for LLM security detectors that addresses systematic weaknesses like per-dataset threshold tuning and undisclosed operating points. The framework uses cross-validation across 16 benchmarks, selects a single global operating point, and includes multiple diagnostics for generalization.