@vikhyatk: Got sick of hand-tuning GPU kernels, so we built a compiler. Photon 2.0 compiles Moondream, Qwen 3.5, and Gemma 4 into …

Got sick of hand-tuning GPU kernels, so we built a compiler.

Photon 2.0 compiles Moondream, Qwen 3.5, and Gemma 4 into megakernels: the entire forward pass in one GPU program.

For each model we define the forward pass using a tracer DSL, that maps out data flows and dependencies for the compiler. Here’s a simplified example showing what it looks like for a dense Moondream layer.

The compiler then explores various ways to schedule the work on the GPU SMs. Actually compiling the megakernel is slow, so we built a CPU-only cost model that tries to simulate execution time for a given schedule. The top candidates then get compiled to find the best one.

Here’s an illustration of the schedule it generates for the dense Moondream layer.

It starts the MLP’s FC2 projection while attention is still scanning the KV cache. By the time attention finishes, most of the MLP work is already done, so the final output projection can join both results immediately.

lol ok i’ll get it out next week

at least this way you know the post wasn’t AI slop lol

i regret missing out on mcnuggets. could’ve punted launch by a day…

sanity? lost that a long time ago lol

There’s less headroom at batch size 1 / single GPU as the model gets bigger, since synchronization cost gets amortized by large matmuls. But in multi-GPU inference (i.e., even bigger models) and higher concurrency there’s still a fair amount of room.

ty, will do!

soon! added to the model backlog

still in the arena lol

which GPU?