Tag
This paper explores the trade-off between factual knowledge acquisition and capability retention in language models when using QLoRA, showing that higher-rank QLoRA improves fact learning but may degrade out-of-domain performance.
The author shares personal insights on the relationship between AI and infrastructure, arguing that while AI amplifies execution capabilities, it cannot replace human judgment on complex trade-offs and organizational issues. The author illustrates this with their own experience developing the cocoon/sandbox project using models like Fable 5.