Tag
FLEET introduces a memory mechanism to text generation in large language models, using logits entropy to enhance trajectories, resulting in improved accuracy and a 3x speedup, particularly on coding tasks.
This pull request optimizes llama.cpp by avoiding unnecessary copying of logits during prompt decode in multi-token prediction, improving inference performance.