training

This paper introduces techniques to manage memory peaks in training large Mixture-of-Experts models with long context lengths, including Pipelined LLEP, Ring-DTP, SCO, and OffloadStreamAdamW, which enable fixed GPU working sets and improve throughput up to 10.4x.

The article reveals that OpenAI's terms allow it to use paid users' interactions for training despite opt-out options, due to a loophole with reasoning tokens not being classified as user-owned output.

Google Search's AI features can help runners prepare for races by offering personalized training plans, custom playlists, and gear recommendations.

The article reflects on the historical lack of knowledge sharing and training in Digital Forensics and Incident Response (DF/IR), highlighting personal experiences and efforts to improve processes through automation and documentation.

The author is experimenting with an adaptive memory governor for PyTorch to prevent CUDA OOM errors on 8GB GPUs, sharing code and seeking community feedback.

An experiment where two segmentation models were trained using GPT-6 Astra Ultra as an orchestrator without human labels, with lazy prompts due to time constraints, showing predictions on held-out videos.

4000 NVIDIA GB200 GPUs arrived in Texas for the Horizon TACC cluster, forming the largest academic supercomputer, with plans to train open AI models.

A lightweight TTS implementation, replicating Audio8's training with 2000 hours of data, achieving a SIM metric of 0.72 in under 10 hours of training on H200.

This is a massively revised Machine Learning Engineering open book, updated with the latest hardware specifications and examples, providing practical guidance for training and fine-tuning large language models and multi-modal models.

This blog post introduces how to train and finetune multi-vector embedding models using the Sentence Transformers library, showcasing its v6.0 update with a new MultiVectorEncoder type and demonstrating superior performance on medical retrieval tasks.

This paper proposes a unified dynamical framework for training, learning, and inference in neural systems.

This article discusses a high-priced course aimed at preparing individuals for emerging AI jobs that are not yet fully defined, highlighting the growing demand and uncertainty in the AI job market.

Training of the Marin 535B-A23B AI model has begun with an open process, involving pretraining and midtraining on 18.75T tokens using GB200 NVL72 hardware over about 3 months.

Runtime by Modal is a conference event for AI and ML engineers to discuss running AI in production, featuring talks on inference, training, and agents.

Matt Pocock shares a template letter for developers to convince their bosses to invest in his AI Hero course, which teaches a structured engineering process for using AI coding assistants effectively.

A tweet announces the live training of a 535B parameter (23B active) large language model, with links to follow the process on Weights & Biases and GitHub.

An experiment training three from-scratch LLMs with the same GRPO recipe yielded inconsistent results, with GRPO degrading performance in some models, particularly the middle-sized one, and no clear relationship to scale.

LEGO-RL presents a framework to bridge native coding-agent harnesses with scalable policy-gradient reinforcement learning, improving performance on benchmarks like SWE-bench.

The article discusses challenges in training browser-agent models for sequential decision-making, such as error recovery and memory, and seeks input on optimizing training objectives. The author mentions working on the 'mako' model at tinyfish and invites community feedback.

Registrations are open for an 8-week AI Engineering Cohort focusing on RAG & Agents, aimed at software engineers upskilling into AI roles.