Tag
A new technique called JEVfire enables existing LLMs like Qwen to behave more like Jev by modifying decision-making processes without retraining, resulting in significantly faster JSON generation and enabling local AI agents to run efficiently on consumer hardware.
The author presents a method using GBNF grammars in llama.cpp to constrain local AI model outputs, ensuring valid JSON. A compiler is built to convert tool schemas into grammar rules, with per-turn narrowing for relevant tools.
This paper explores distilling a large reasoning teacher model (8B) into a small student model (0.6B) for on-device structured text enrichment, achieving substantial speedup while recovering significant quality. The study finds that the reasoning nature of the teacher, rather than its scale, drives improvement in summary quality, but a same-size instruction teacher yields more faithful outputs on certain articles.
This paper proposes Dynamic Infilling Anchors (DIA), a training-free method for diffusion large language models that dynamically estimates end-anchor positions to enforce format constraints (e.g., parseable JSON, reasoning templates) while avoiding the rigidity of fixed-span approaches. Experiments show significant zero-shot gains on GSM8K and MATH benchmarks.
Outlines is a Python library that guarantees structured outputs (JSON, Pydantic models, etc.) from LLMs during generation, working across multiple model providers to eliminate parsing errors.