Tag
Announcement of a blog post explaining the design of BarunLM, a 35M-parameter language model, covering its architecture, training recipe, and dataset preparation, with a focus on efficiency and performance gains over other sub-100M models.
Proposes Distance-Adaptive Representation (DAR) which reduces key-value dimensionality for distant tokens while preserving full dimensionality for nearby tokens, improving KV cache efficiency without performance loss.
Explains sparse attention in transformers, which reduces computational complexity by attending only to a subset of tokens using local or learned attention patterns.