More context :

A single Waymo car uses 4 H100 GPUs just to drive itself. That is about $200k worth of compute inside one car.

Right now, the Nvidia Orin self-driving modules used in Mercedes pack, and Tesla FSD use 64GB of LPDDR5X RAM.

Most people think better tech means we will need less memory over time. The exact opposite is happening.

Tesla’s upcoming AI5 chip will jump to 96GB, and the AI6 chip is going straight to 144GB.

Even your smart fridge and smart TV need RAM now. Soon, literally every physical object around us will have memory built into it. And as robotics takes off, those memory numbers are going to get much bigger.

There is no way around the hardware bottleneck: If AI robots run on the cloud, you need massive data centers full of memory. If AI robots run on-device, you have to pack huge amounts of RAM directly into the machine.

Either way, the world is going to need an insane amount of memory.

This is not a problem we can fix easily.

Waymo lifts the lid on the ‘brain’ powering its robotaxis

Source: https://www.theverge.com/transportation/982653/waymo-brain-computer-chip-robotaxi-hardware-suppliers What’s in the trunk of your car? Shopping bags? A spare tire? Jumper cables? What about a high-powered computer capable of performing up to one quadrillion operations a second? Only if you’re Waymo.

For the first time, Waymo revealed key details about the heavy compute “brain” housed in the trunk of its robotaxis, including chip architecture, processor specs, and internal component details. The details,published in a blog post today, also includes a list of hardware suppliers that Waymo uses to build the computers that power its driverless fleet.

It’s a revealing look at the world’s leading player in driverless technology. Waymo is currently operating around 4,000 vehicles in over 10 cities, conducting approximately 500,000 paid trips a week. The company claims that its technology significantly reduces traffic crashes and injuries as compared to human drivers. And while a lot of the work to run its vehicles relies on heavy training and simulation that runs in the cloud, a good portion also takes place in the vehicles themselves.

Waymo’s robotaxis, festooned with cameras, lidar, and radar, take in massive streams of raw sensor data. That data is processed by artificial intelligence to instantly understand what’s happening around the car, turning that information into safe driving commands in milliseconds. Its computers, which are installed in the trunks of its vehicles, are built to act like its eyes, ears, and brain, to process a full 360-degree view and make split-second decisions without any human backup.

“Compute systems for autonomous driving handle highly diverse workloads,” Satish Jeyachandran, VP of Engineering, and Daniel Rosenband, Compute Lead, write in the blog post. “At Waymo, we are designing a state-of-the-art system that would be considered impressive for a data center, with the added complexity of an in-vehicle operating domain and real-time requirements.”

Jeyachandran and Rosenband describe three core principles guiding the design of its compute: responsive, ruggedized, and redundant. The stack needs to be able to respond to sensor data as quickly and with as little latency as possible. This requires massive amounts of compute power, which Waymo has scaled 20 times in the past eight years. It also needs to be able to handle all the bumps and cracks in the road, as well as extreme temperatures. And it basically needs to be able to run two systems in parallel, in the event that one fails. These three principles are “non-negotiable,” the executives write.

It also needs to not take up the entire trunk. After all, there are passengers and their luggage to consider. And it needs to run nearly silent, so as not to compromise the rider experience. And while Waymo builds its compute stack itself, it also relies on a network of third-party suppliers to provide the components it can’t build itself.

Waymo’s effort to explain how its multi-sensor system can fuse data in real time is especially noteworthy considering Tesla CEO Elon Musk’s dismissal of these systems as inherently faulty. Musk has called lidar a “crutch” and “a fool’s errand,” arguing that sensor fusion between cameras, radar, and lidar introduces dangerous “sensor contention.” He also contends that when differing sensor data disagree, figuring out which signal to trust creates deadly ambiguity rather than safety.

But Waymo argues that its robust and redundant system is what enables it to deploy its vehicles at scale. “We built an [machine learning]-primary architecture to run advanced neural networks at minimal latency,” the executives write. “To manage critical non-ML tasks like orchestration, data movement, and logging while maximizing time for ML computation, we pair our ML technologies with the best CPUs, GPUs, and accelerators. The result is a balanced, heterogeneous system.”

In the blog post, Waymo introduces its custom silicon chip, a 5-nanometer ASIC, built to handle the massive firehose of incoming data from its sensors. Instead of sending raw, messy data straight to the car’s main brain, the chip sits at the front lines cleaning up the data, combining the sensor inputs, and running fast AI checks right as the information comes in. The chip’s dedicated front-end processing provides 1,000 TOPS (trillion operations a second). And because it was built specifically for Waymo’s autonomous driving operations, it fits neatly in the overall stack, helping keep reaction times minimal.

Consider this scene below. Waymo’s executives claim its system can more quickly process high-fidelity data from its 13 high-res cameras “simultaneously and in real-time” to deliver improved low-light perception. In that way, it can see a lot more stuff in the dark (the image on the right) that traditional cameras often miss (the image on the left).

Image: Waymo

Jeyachandran and Rosenband conclude the blog post by running through a list of hardware suppliers who are helping build Waymo’s cutting edge system: AMD, Micron, Nvidia, Samsung, SanDisk, Socionext, and TSMC. Rather than try to build its own compute system from scratch, the company is instead focusing its internal efforts on real-time sensor fusion and front-end machine learning, while leaning on the global supply chain for everything else.

It’s a revealing glimpse of the technology that Waymo uses to power its fleet of robotaxis. But the company doesn’t get into one important aspect: How much does all this cost? Some estimates put it at$20,000 to $25,000 per vehiclefor its sixth-generation hardware. While that’s a huge decrease from the fifth-generation’s estimated cost of $100,000 to $125,000 per vehicle, it still shows how Waymo’s continued expansion could be hampered by high hardware expenses.

Building a brain to run an entire fleet of driverless cars is a monumental achievement, no question. But building a business that can make a profit is another challenge altogether.

Follow topics and authorsfrom this story to see more like this in your personalized homepage feed and to receive email updates.

• Andrew J. Hawkins

Jun Song (@jun_song): Hate to break it to you, but the global hardware shortage has not even started yet.

Right now, only 1% of humanity is burning any real amount of AI tokens. The rest are just using free accounts once in a while to ask dumb questions like “Who would win in a fight between a baby