Roadmap: How to become a GPU/CUDA engineer with AI

Outlook

NVIDIA software engineer compensation in 2026 ranges from $176k for IC1 to $1.04M for IC7, with median total comp at $390k (Levels.fyi). Inference optimization specialists - the ones who take a trained model and make it run 3x faster on the same GPU - sit in the 205k-331k band at NVIDIA, and $400k+ at OpenAI. The skill is rare. The demand is not slowing. And in 2026 the path to it has changed materially.

This article maps the optimal AI-augmented path through one of the hardest, best-paid technical specializations in software. AI tools can now profile your kernels, explain warp-level semantics, and walk you through a paper implementation in an afternoon. The tradeoff: AI is weaker on CUDA than on Python because the training data is thinner. Some sequences still require months of solo experimentation. Others have been genuinely shortened.

The lens: not “how to get rich at NVIDIA,” but what learning a hard, well-paid technical specialization actually looks like in 2026.

The numbers you need to see first

Real compensation by tier

• IC1 at NVIDIA (entry-level): base around $130-150k, total comp $176k+ (Levels.fyi, 2026)

• Mid-level GPU engineer: base $140-170k, total comp commonly $200-280k

• Senior at NVIDIA / Google / Meta: total comp $300-500k

• Inference optimization specialists (rare skill): NVIDIA $205-331k, OpenAI $400k+

• Staff and principal at top firms: 500k-1M+

• The IC7 ceiling at NVIDIA: $1.04M total comp

Caveat that matters: the high numbers are total compensation heavily weighted toward stock, not base salary, and they require real seniority. A junior who just finished this roadmap walks into the $130-180k band. The path to the high numbers is several years of shipping real optimization work on real systems.

What changed in 2026 that makes this relevant now

• LLM inference is now the single largest GPU workload in the industry. Every AI lab needs people who can make models run faster.

• Inference optimization has become its own job category. OpenAI, Anthropic, NVIDIA, Meta have dedicated postings for “model inference engineer” or “performance engineer”.

• The training stack has consolidated. PyTorch + Triton + CUDA is the dominant pattern. Knowing this trio in 2026 is what knowing C++ was in 2005.

• New hardware (Hopper H100, Blackwell B200) is rolling out faster than people can learn to use it. The skill gap is widening, not closing.

• Open-source kernel work has become a real recruiting pipeline. Merged PRs to vLLM, PyTorch, FlashAttention, Liger-Kernel get noticed.

This is one of the few specializations where the supply of qualified engineers has not caught up to demand, three years into the AI boom. The market is telling you something. The reason most people do not move into this niche is that the path is genuinely harder than web dev. AI now makes the path achievable for the kind of person who can absorb dense technical material quickly.

A real expectation about timelines

This roadmap assumes you absorb dense technical information well, you can do 20+ hours per week of focused work, and you ship code rather than passively consuming tutorials. With those three conditions, here is the honest tier map:

• Year 1: you can write CUDA and Triton kernels that work and that you can profile yourself. Entry-level GPU engineer roles open up. Salary band $130-180k.

• Years 2-3: you can take a real production kernel, identify the bottleneck without guessing, and close most of the gap to peak hardware throughput. Mid-level. Salary band $180-280k.

• Years 4-7: you can architect optimization projects, write papers or open-source kernels that other people use, lead initiatives. Senior to staff. Salary band 300k-700k+.

People who tell you “one year to production GPU engineer” are either selling you something or talking about someone who already had years of low-level systems programming background. From scratch, on the edge of what is achievable, the first tier takes 12 months. Most people take 18.

The good news: there is no PhD gate. NVIDIA, OpenAI, and Anthropic hire from open-source contribution records. If you have merged kernels in vLLM or PyTorch, you do not need a PhD to be looked at.

The ladder

Each step is a prerequisite. The biggest mistake is jumping to Step 4 (optimization) without Step 1 (hardware fundamentals). You can write a kernel that runs without understanding the roofline model. You cannot optimize one.

Step 1: Hardware fundamentals (1-2 months)

Start not with CUDA, but with why GPUs exist at all.

What to learn:

CPU vs GPU architecture

• SIMD vs SIMT execution models

• Out-of-order vs in-order execution

• Why GPUs win on throughput, lose on latency

• Branch prediction (CPU) vs warp divergence (GPU)

• Memory bandwidth as the dominant constraint in modern compute

Memory hierarchy on modern hardware

• L1, L2, L3 cache sizes and access latencies

• DRAM bandwidth vs cache bandwidth (orders of magnitude difference)

• HBM vs GDDR memory (datacenter GPUs vs consumer)

• Cache lines, false sharing, NUMA on CPU side

• Why locality matters: temporal and spatial

GPU execution hierarchy

• Threads, warps (32 threads on NVIDIA), blocks, grids

• Streaming Multiprocessors (SMs) and how blocks map to SMs

• Warp scheduling and what “occupancy” actually means

• Hardware difference: Ampere (A100) vs Hopper (H100) vs Blackwell (B200)

• Tensor cores: what they are, when they activate

The roofline model

• Arithmetic intensity (FLOPs per byte)

• The ridge point: where compute-bound becomes memory-bound

• How to compute the roofline for your specific GPU

• Practical use: knowing without profiling whether optimization will help

Why this layer is non-negotiable: every optimization decision in the rest of the roadmap comes back to “am I compute-bound or memory-bound.” If you cannot answer that with a profiler trace and a calculation, you will guess. Guessing wastes weeks.

Resources:

• Hennessy and Patterson, Computer Architecture, GPU and parallelism chapters

• NVIDIA Hopper architecture whitepaper

• NVIDIA Blackwell architecture whitepaper

• The roofline model paper (short, worth reading twice)

Practice:

• Write a C program doing array operations, measure actual memory bandwidth

• Compare to the theoretical bandwidth of your GPU

• Plot a simple roofline for one kernel

Where AI helps in this step:

• Explaining whitepaper sections you do not understand. The Hopper paper assumes you know what tensor cores are. Claude can walk you through it.

• Generating practice problems calibrated to your level

• Sanity-checking your roofline calculations

Where AI does not help:

• Building real intuition about memory hierarchy. That comes from running code, watching it underperform, and figuring out why. AI can describe locality. It cannot make you feel it.

Step 2: CUDA basics (2-3 months)

Now CUDA itself.

What to learn:

CUDA programming model

• Kernel launch syntax: <<<grid, block>>>, dynamic shared memory size, stream

• Thread indexing: threadIdx, blockIdx, blockDim, gridDim

• 1D, 2D, 3D thread organization (when each makes sense)

• Streams and concurrent kernel execution

• Events for timing and synchronization between streams

Memory hierarchy in CUDA

• Global memory: largest, slowest, where your data lives

• Shared memory: per-block, fast, ~100x faster than global, but tiny (48-228KB)

• Registers: per-thread, fastest, limited (causes register spilling if abused)

• Constant memory: read-only, broadcast-friendly

• Local memory: misleading name, actually lives in global memory (spilled registers)

• Texture and surface memory (mostly graphics, occasionally useful in ML)

Synchronization

• __syncthreads() and what it actually does

• Warp-level primitives: __shfl_sync, __ballot_sync, __any_sync

• Cooperative groups API (modern alternative)

• Atomics: atomicAdd, atomicCAS, when they kill performance

• Memory fences and acquire-release semantics

Coalescing and bank conflicts (the two beginner killers)

• Coalesced vs uncoalesced global memory access patterns

• Why a 32x slowdown is possible from one bad pattern

• Shared memory bank conflicts: 32 banks, what causes 2-way, 4-way, 32-way conflicts

• How to detect both with ncu

Modern tooling

• Nsight Systems (nsys) for timeline profiling: kernel launches, memory copies, gaps

• Nsight Compute (ncu) for kernel-level analysis: occupancy, memory throughput, instructions

• How to read ncu’s “rules” output (it tells you what is wrong)

• Sampling profilers vs instrumenting profilers

One critical note on tooling: nvprof is dead. It was deprecated years ago and does not support modern GPUs. If a tutorial tells you to use nvprof, it is old. Use nsys and ncu.

Resources:

• Programming Massively Parallel Processors (PMPP), Kirk and Hwu. The standard CUDA textbook. If you read one book, read this.

• GPU MODE (formerly CUDA MODE), the reading group run by Andreas Kopf and Mark Saroufim from the PyTorch team. They work through PMPP chapter by chapter. Active Discord. This is the closest thing to a community standard right now.

• NVIDIA CUDA C++ Programming Guide - official, complete, use as reference

• NVIDIA CUDA samples on GitHub

Practice:

• Vector addition kernel (trivial, but write it)

• Naive matrix multiplication

• Profile it with ncu, find why it is slow, fix coalescing

• Compare your numbers against cuBLAS for the same operation

Result after this step: you can write a kernel that runs, profile it, and explain why it is fast or slow.

Where AI helps:

• Reading nsys/ncu output. Paste a profile trace, ask “what is this telling me about my bottleneck.”

• Explaining warp divergence with concrete examples instead of abstractions

• Debugging segfaults in CUDA kernels (AI is genuinely good at this when given the kernel source plus error)

• Generating progressively harder practice problems on coalescing patterns

Where AI is weaker:

• Suggesting non-obvious optimizations. AI knows the textbook patterns. It does not know what worked on your specific hardware at your specific occupancy.

• Hardware-specific tuning for the newest GPUs (Blackwell came out faster than training data caught up)

Step 3: Triton and production libraries (3-4 months)

This is the part most old roadmaps miss, and it is now the most important step.

Most GPU work in ML is no longer raw CUDA. It is Triton, OpenAI’s language for writing GPU kernels in Python-like syntax. PyTorch 2.0 generates Triton under the hood for torch.compile. vLLM has a Triton attention backend that achieves 100.7% of FlashAttention-3 performance on H100 with a single portable kernel. If you want to work on ML inference or training in 2026, Triton is not optional.

What to learn:

Triton fundamentals

• @triton.jit decorator and compilation model

• tl.program_id, tl.arange, tl.load, tl.store

• Block-level vs thread-level thinking (different mindset than CUDA)

• Masking for boundary conditions (instead of explicit bounds checks)

• Autotuning with @triton.autotune

• When to use BLOCK_SIZE constants vs dynamic block sizes

cuBLAS

• gemm, gemv, batched gemm

• Performance hierarchy: cublasGemmEx > cublasSgemm > naive

• Why your custom matmul will rarely beat cuBLAS (and when it will)

• Workspace management

cuDNN

• Convolution APIs (forward, backward filter, backward data)

• Algorithm selection (heuristic vs benchmarking)

• Tensor descriptors and layout (NCHW vs NHWC)

• Why direct cuDNN use is rare in 2026 (PyTorch wraps everything)

TensorRT

• ONNX as input format

• Builder, network, engine, context (the lifecycle)

• INT8 calibration for quantization

• Plugin API for custom operators

• When TensorRT wins (production deployment) vs loses (research/training)

CUTLASS

• Hierarchical GEMM design (block, warp, thread tiles)

• When to use CUTLASS vs cuBLAS (CUTLASS gives you custom epilogues)

• 3.x API and CuTe (the new tensor abstraction)

• Fused epilogue patterns (matmul + bias + activation in one kernel)

Embedding kernels in Python

• PyTorch custom op API (torch.library)

• pybind11 basics for C++/CUDA interop

• Compiling extensions with setup.py vs JIT compilation

• Handling autograd for custom backward passes

Also worth knowing about (2026 additions):

• FlexAttention in PyTorch - for custom attention masks without writing CUDA

• Liger-Kernel - efficient Triton kernels for LLM training, used in production

• Helion - new DSL from PyTorch team, higher-level than Triton

Don’t waste time on:

• Reading entire cuDNN docs front to back. Reference as needed.

• Becoming a TensorRT expert in a month. Deep rabbit hole, come back later.

Resources:

• Triton official tutorials - start here, they are excellent

• NVIDIA deep learning library docs (cuDNN, TensorRT)

• CUTLASS GitHub examples

• nano-vLLM - 1,200 lines that reimplement the core of vLLM in Python and Triton. Read the entire codebase in an afternoon, understand PagedAttention and continuous batching.

Practice:

• Rewrite your matrix multiply in Triton, compare to your CUDA version and to cuBLAS

• Write a fused kernel in Triton (matmul + activation in one pass)

• Take an ONNX model, run it through TensorRT, measure the speedup

• Read nano-vLLM in a weekend, understand how PagedAttention works at the kernel level

Result: you can pick the right tool for a problem instead of writing everything from scratch.

Where AI helps a lot:

• Translating CUDA to Triton (the patterns map, AI knows them)

• Explaining what a block-level operation actually does at the hardware level

• Writing first-draft Triton kernels you then optimize

• Sanity-checking your kernel against PyTorch eager-mode reference

This is probably the step where AI gives the largest acceleration. Triton is Python-like, AI is strong on Python, and the optimization patterns map cleanly from CUDA literature.

Step 4: Deep optimization (4-6 months)

This is where it gets hard. This is what separates engineers from people who know CUDA.

What to learn:

Warp-level programming

• Warp shuffle instructions: __shfl_sync, __shfl_up, __shfl_down, __shfl_xor

• Warp-level reductions (replace shared memory + __syncthreads pattern)

• Warp specialization (different warps do different work in same block)

• Warp divergence: cost and how to detect it in ncu

Occupancy

• Theoretical vs achieved occupancy

• The myth: “higher occupancy = better performance” is often wrong

• Register pressure and how to reduce it (-maxrregcount, launch_bounds)

• Shared memory usage and its effect on occupancy

• When low occupancy is actually optimal (memory-latency hiding via ILP)

Tensor cores (where modern performance lives)

• mma instructions and what tiles they consume (16x16x16, 8x8x4, etc)

• WMMA API vs PTX-level mma vs CUTLASS abstractions

• FP16, BF16, TF32, FP8, FP4 (Blackwell): when each is appropriate

• Why feeding tensor cores requires specific memory layouts

• Async copy (cp.async on Ampere+) for hiding memory latency

Reduction patterns

• Tree reduction in shared memory (the classic)

• Warp shuffle reduction (faster, no shared memory)

• Block-level vs grid-level reductions

• Parallel prefix scan (Blelloch vs Hillis-Steele)

• Segmented reduction for jagged data

Kernel fusion

• Why fusion matters: kill memory roundtrips, not compute

• Vertical fusion (chain of operations) vs horizontal fusion (parallel operations)

• Producer-consumer patterns and shared memory as a buffer

• Why FlashAttention is the canonical fusion example

• Limits of fusion: when it kills occupancy or register usage

Atomics and their real cost

• atomicAdd hardware path (L2 vs DRAM atomics, depending on architecture)

• Contention: 32 threads atomicAdd-ing same address = serialized

• Histogram patterns and how to avoid global atomics

• Warp-level aggregation before global atomic

FlashAttention is worth studying closely. It is the canonical example of how rethinking memory movement, not adding compute, produces massive speedup. Read the paper, then read a clean implementation. It will teach you more about real optimization than a month of tutorials. The FlashAttention-3 paper extends this to Hopper-specific optimizations.

Resources:

• NVIDIA Kernel Profiling Guide

• Simon Boehm’s “How to optimize a CUDA matmul kernel” worklog. This single post takes a naive matmul to near-cuBLAS performance step by step. Going through it yourself is worth a month of tutorials.

• GTC talks - search kernel optimization, tensor cores

• FlashAttention paper and reference implementation

• High-performance kernels on GitHub: CUTLASS, PyTorch ATen, vLLM

Practice (this is the core of the whole year):

• Optimize your matmul until it is within a reasonable factor of cuBLAS

• Implement a tiled, shared-memory reduction

• Implement a simplified attention kernel with fusion

• Use ncu to find and remove each bottleneck, one at a time

• Replicate one optimization from FlashAttention on a toy problem

Result: you see the difference between code that works and code that is optimized, and you can close that gap yourself.

Where AI helps:

• Reading optimization papers. FlashAttention has dense notation. AI can walk you through it.

• Suggesting things to try when you are stuck on a specific bottleneck

• Code review of your tensor core usage (AI knows the constraints, like FP16 alignment requirements)

Where AI is genuinely weaker here:

• Truly novel optimization ideas. AI knows what is in papers. Real optimization at this level often requires inventing something the literature does not have.

• Hardware-specific tuning for the newest GPUs. AI knowledge here is roughly 6-12 months behind cutting edge.

This is the step where mentorship matters most. If you can pair with someone senior, even informally, this is the time. AI alone gets you 70% of the way. The last 30% historically came from senior engineers and that has not fully changed.

Step 5: Specialization (6-8 months)

Pick one direction. All three are in demand.

Inference (highest demand in 2026)

What to learn:

TensorRT in depth

• Network builder API and layer-by-layer construction

• INT8 calibration: entropy calibrator, MinMax calibrator

• Engine serialization and deployment

• Multi-stream execution for throughput

• Plugin development for custom ops

Quantization techniques

• Post-training quantization (PTQ) vs quantization-aware training (QAT)

• INT8 symmetric vs asymmetric

• FP8 on Hopper (E4M3 vs E5M2 formats)

• FP4 on Blackwell (new in 2026, significant inference gains)

• GPTQ, AWQ, SmoothQuant for LLM-specific quantization

Continuous and dynamic batching

• Static batching: why it wastes GPU cycles

• Continuous batching: how vLLM does it

• Request scheduling and KV cache eviction policies

• Speculative decoding for latency reduction

Kernel fusion patterns for serving

• LayerNorm + linear fusion

• Attention QKV projection fusion

• RoPE applied inline with QK

• Fused MoE expert routing

vLLM internals (study this codebase deeply)

• PagedAttention: block table indirection, no fragmentation

• KV cache management as paging system

• Continuous batching scheduler with cache-pressure awareness

• Prefix caching for shared system prompts

Training

What to learn:

Backward pass kernels

• Why backward is often slower than forward (extra memory traffic)

• Recomputation vs storing activations

• Implementing custom autograd Functions in PyTorch

• Gradient accumulation across micro-batches

Collective communication (NCCL)

• all-reduce, all-gather, reduce-scatter, broadcast

• Ring all-reduce vs tree all-reduce: when each wins

• NVLink topology and how it affects collective performance

• Overlap of compute and communication

Mixed precision and FP8 training

• Loss scaling for FP16 (why it’s needed)

• BF16 vs FP16: when to use which

• FP8 training: transformer engine, scaling factors per tensor

• Gradient clipping in mixed precision

Memory-saving techniques

• Gradient checkpointing: trade compute for memory

• Activation offloading to CPU/NVMe (DeepSpeed Zero-Infinity)

• Optimizer state sharding (ZeRO stages 1, 2, 3)

• Selective recomputation (recompute only attention)

Distributed training patterns

• Data parallel (DP, DDP, FSDP)

• Tensor parallel (Megatron-style)

• Pipeline parallel (GPipe, PipeDream, 1F1B)

• Expert parallel for MoE models

• 3D parallelism: combining all three

HPC / Scientific Computing / Graphics

What to learn:

For graphics:

• Texture sampling, anisotropic filtering, mipmapping

• Rasterization pipeline (vertex, fragment, depth tests)

• Compute shaders vs traditional graphics pipeline

• Vulkan compute or DirectX 12 compute basics

For HPC/scientific:

• Stencil computations (PDE solvers, weather, fluid dynamics)

• Particle systems (N-body, molecular dynamics)

• Sparse linear algebra (CSR, CSC, COO formats)

• Multi-GPU domain decomposition

This path is more games and HPC than ML. Lower competition than ML, more stable demand, slightly lower ceiling.

Practice:

• For inference: take a real model, optimize end to end, measure tokens per second before and after. Publish the writeup.

• For training: implement forward and backward for a small network, profile multi-GPU if you can afford it

• For HPC: contribute to a scientific kernel project, measure your contribution’s impact

Result: you understand how production systems in your chosen area actually work, and you have one writeup that demonstrates this.

Where AI helps a lot in specialization:

• Understanding what a specific paper actually contributes vs prior work

• Reading codebases like vLLM (large, complex, AI can summarize layers and explain interactions)

• Generating eval harnesses for your optimizations

Step 6: Open-source contributions and the offer (8-12 months)

At this stage you learn by shipping, not reading.

What to do:

• Contribute to open source. The strongest options in 2026: PyTorch, vLLM, Triton, TensorRT-LLM, SGLang, Liger-Kernel. Real merged PRs are the strongest possible portfolio.

• Build public artifacts: a repo of optimized kernels with benchmarks and writeups explaining before and after

• Follow NVIDIA developer blogs and optimization papers as they come out

• Watch GPU MODE lectures as they happen

Where to apply:

• NVIDIA

• OpenAI

• Anthropic

• Google DeepMind

• Meta AI Infrastructure

• Inference startups: vLLM-adjacent companies, Modular, Groq, Together AI, Fireworks AI

• HPC and quant firms (the latter pay well and rarely advertise)

Mistakes that slow you down

• Using nvprof. It is deprecated. Use nsys and ncu.

• Skipping Triton because it “feels like cheating”. It is what a lot of the job actually uses.

• Optimizing without profiling. You will spend weeks tuning the wrong thing.

• Forgetting the CPU side. Host-to-device data movement is the bottleneck more often than the kernel.

• Specializing too early. Build the foundation, then go deep.

• Reading instead of building. After month 3, every concept should be paired with a kernel you wrote.

• Believing “one year to senior”. Year 1 gets you in the door. Senior takes years of shipping.

• Using AI as a crutch. It accelerates learning but does not replace doing the work.

Where to get a GPU

You do not need to own a GPU to start.

• Google Colab gives free GPU time, enough for the first months

• For serious work, rent by the hour: vast.ai, RunPod, Lambda

• A single recent consumer card (used RTX 3090 or 4090) is enough to learn everything in this roadmap

• You do not need a data center GPU until you are working on multi-GPU problems

An hour of rented compute costs less than a coffee. Hardware is not the reason not to start.

What separates an engineer from someone who knows CUDA

Someone who knows CUDA can write a kernel that works, call existing libraries, and sometimes find a bottleneck with a profiler.

An engineer can look at a kernel and predict the bottleneck before profiling. Can write code that hits a large fraction of theoretical peak instead of a small one. Can reason about the memory versus compute tradeoff without thinking about it. Can debug a production regression in hours instead of days.

The gap is years of deliberate practice on real problems. The first year of this roadmap gets you to “knows CUDA”. The second and third years get you to “engineer”. The years after that are where compensation starts to look the way the headlines suggest.

Where to start today

• Install the current CUDA Toolkit

• Write a vector addition kernel

• Profile it with ncu, not nvprof

• Change something, watch the profile change, understand why

• Read PMPP chapter 1

• Subscribe to GPU MODE on YouTube

A month of this tells you whether you want to continue. If you do, the path runs for years and the demand is not slowing down.

Where to start tonight (free resources)

Open-access materials you can work through without paying:

• GPU MODE YouTube lectures - the closest thing to a community standard course

• NVIDIA CUDA C++ Programming Guide - official, complete

• Triton tutorials

• Simon Boehm matmul worklog - the best practical optimization writeup

• FlashAttention paper

• nano-vLLM - readable in an afternoon, teaches more than weeks of tutorials

• CUDA samples

• NVIDIA developer blog

Compute:

• Google Colab for free GPU hours

• vast.ai and RunPod for cheap rentals

Sources

Compensation data:

• Levels.fyi - NVIDIA Software Engineer - $176k IC1 to $1.04M IC7, median $390k

• Pin 2026 AI Compensation Benchmarks - $173k median, $269k 90th percentile for AI/ML engineer

• The 56% Premium - Let’s Data Science - inference optimization $205-331k at NVIDIA, $400k+ at OpenAI

• ZipRecruiter CUDA Engineer Salary - postings ranging 88k-333k

Industry context:

• vLLM Triton Attention Backend Deep Dive - portable Triton kernels matching FlashAttention-3

• PyTorch FlexAttention production guide - custom attention without CUDA

• nano-vLLM walkthrough - 1,200 lines that teach you vLLM

• LLM Engine Room Part 2 (Medium) - what production inference actually looks like

• Breaking Into AI in 2026 (dataexec.io) - hiring landscape at Anthropic, OpenAI, Meta