Tag
该论文研究 LoRA 适配器初始输入子空间方向的重要性为何依赖于数据 regime,提出基于激活 PCA 的 PivotLoRA 初始化方法,并在少样本 BERT/Llama 分类及 GSM8K、StrategyQA 任务上验证其优于原版 LoRA。
前Meta研究员Stephen Roller在法庭证词中承认在职期间通过BitTorrent下载并分发了盗版Books3数据集,该数据集被用于训练Llama模型,尽管Meta此前否认通过BitTorrent获取该数据集。Meta回应称该下载可能是为个人用途,而原告计划要求法院进一步查明Meta获取和使用该数据集的详情。
提出 CADENCE(Context-Adaptive Decoding)方法,通过高斯预测头与学习式路由自适应选择每次解码的跨度长度,在 Llama-3.1-8B-Instruct 上实现最高 1.3× 的生成吞吐提升。
该论文研究如何将一个模型表示上学到的量化校准增益复用到同一预训练模型的其他表示(如浮点与INT3量化)上,无需接收端优化即可保留大部分文本损失改善,量化了校准更新的可迁移性与接收端专属训练的剩余收益。
Mark Zuckerberg discusses why Meta's Llama development faltered after Llama 3, explaining that Llama 4 was staffed like an Instagram feed/ads team with thousands of engineers rather than a small frontier-science team, and reflects on how leaders respond when things go wrong.
The paper introduces attention manifolds — learned 2D B-spline surfaces that modulate value dimensions in transformer attention — showing 2–2.5 point perplexity improvements on LLaMA 3.2 with just 0.3% parameter overhead, plus mechanically editable surfaces enabling steering and attention walls for safety.
CalDec v1 is a fully open-source, open-weights decision model for real-time, fully-local personal assistants, released with dataset and training recipes. Built by fine-tuning models like Laya and GLiNER, it showed promising results on decision tasks while remaining smaller than general-purpose alternatives like Jev.
A spoken-word piece contrasts the view of AI as incremental technology with the perspective of exponential growth, critiquing figures like Gary Marcus and Yann LeCun while referencing models like LLaMA.
The paper quantifies how improvements in fact-verification scores are partitioned between answer accuracy and evidence quality, using trained DeBERTa checkpoints and LLMs across multiple benchmarks.
This paper investigates the relationship between spontaneous and instructed deception in large language models, specifically Llama-3.1-70B-Instruct, finding asymmetries in detection and causation between the two settings.
A user shares their experience running the ik_llama model locally, praising its speed despite having only 12GB of VRAM.
Muse Spark 1.2 is being open sourced, which is surprising given the focus on Llama 4 Behemoth. The author comments on the unexpected release order.
This paper introduces a game-theoretic approach to fine-tuning language models that optimizes the trade-off between reward and deviating from a reference policy, providing a principled method for setting the KL regularization coefficient.
A Statista chart estimates that approximately a quarter of the training data for Meta's Llama AI model came from books, highlighting the composition of sources used in large language model development.
This paper proposes SPG, a structure-aware patent generation method that jointly predicts claim topology and content during autoregressive decoding using a pointer head and a preference optimization stage, achieving significant improvements in parent link recovery and antecedent consistency on the HUPD-DCG benchmark.
Testing 30 schema constraints across 16 models reveals each provider handles constraints differently, causing random tool call failures. The simple fix: move constraint text into property descriptions instead of relying on prompts.
Yann LeCun defends open source by listing major projects like Linux, Apache, MySQL, and PyTorch, questioning the idea that releasing them was 'dumping'.
This paper identifies a 'countdown subcircuit' in Llama-3.1-70B-Instruct that allows the model to track remaining tokens across various tasks, and demonstrates that this subcircuit is reused across different contexts and even models.
User criticizes Meta's decision to stop releasing Llama models, calling it shortsighted.
Windows Defender flags a version of Llama-b9856 running on CUDA 12.4 as a trojan, raising security concerns.