The head of Nvidia’s autonomous driving division talks about his plan to take on Waymo and Tesla

Every six months or so, Nvidia’s automotive chief, Xinzhou Wu, invites CEO Jensen Huang to go for a ride in a car equipped with the company’s hands-free self-driving system. But only when Wu has “good confidence” in the regime’s leadership capabilities.

Recently, the two went on a drive from Woodside, Calif., to downtown San Francisco in a Mercedes CLA sedan with MB.Drive Assist Pro, a hands-free driver-assistance system designed in part by Nvidia and similar to Tesla’s full self-driving system. The mood was light, even if the traffic was very busy.

“Let me know when it’s in autonomous mode,” Huang Luo said, according to a video of the trip provided to him. Edge“Then I can be less worried about my safety.”

Over the course of the 22-minute video, Huang and Wu’s Mercedes navigates a series of everyday obstacles, such as construction sites, double-parked cars, and narrow lanes past rows of orange cones. The Nvidia system seems quite capable, even though the video is edited and not rendered in real time. (Nvidia spokeswoman Jessica Soares later said there were no disengages during the flight.)

However, it seems no different from My own experience last year riding shotgun with Nvidia executives in a Mercedes with the hands-free driving system activated. I was impressed with the system’s ability to handle traffic lights, four-way stops, two-parked cars, unprotected left turns, and all the pedestrians, cyclists, and scooters that San Francisco can throw at you. If Tesla can do it with a little silicon and a bunch of cameras, it stands to reason that the world’s most valuable company can figure it out too.

“ChatGPT moment for real AI”

After years of work behind the scenes, Nvidia is trying to gain a more prominent leadership position in the autonomous driving space. Not only does it supply chips to companies like Tesla, but it also offers its own AI-powered driving features to partners like Mercedes, Jaguar Land Rover, and Lucid. At CES earlier this yearHuang revealed Albamayoa set of artificial intelligence models, simulation charts and data sets, that can give vehicles Level 4 autonomy, allowing them to drive themselves completely under specific conditions. Huang described the announcement as “ChatGPT’s moment for physical AI.”

In the car with Wu, Huang was less boastful and more introspective, but no less optimistic about the future of technology. “I think the challenge, of course, is Alpamayo, despite his amazing intelligence – and his ability to think through circumstances – we don’t know what he can’t do,” he said. “That’s the challenge, and that’s why our Classic Collection is so incredibly important.”

Hwang boasts that Nvidia’s approach to autonomous driving is “unique” because it combines an end-to-end AI model with a traditional, “classic” human-designed suite. It is believed that it is difficult to verify the integrity of comprehensive models. In contrast, the classic stack follows well-established engineering protocols and processes that make it easy to verify that certain behaviors are safe enough. By combining both approaches, Nvidia’s system can leverage a human-like driving style while maintaining a safety framework grounded in traditional rules of the road.

Huang’s claim of a unique approach to the industry doesn’t quite hold up; Other autonomous vehicle operators also use comprehensive neural networks along with clear safety rules that govern how the vehicle responds. But certainly end-to-end learning, which tends to be more human-like in its leadership and less automated, is becoming more popular. Waymo relies on a hybrid system, while Tesla relies exclusively on end-to-end neural networks.

In an interview, Wu said that all-in-one models are better able to respond to things like speed bumps or lane changes without feeling overly mechanical or robotic. “That’s why it’s really a ChatGPT moment,” he said. “It’s as if your car drives with real confidence…then customers will basically feel more willing to use it.”

Tesla and the rising cost of self-driving

I asked Wu how he thought Nvidia’s approach compared to Tesla’s full self-driving, which has driven more than 8.5 billion miles but has been… Involved in a number of worrying safety incidentsOf these, 23 infections and at least two deaths. Last December, an Nvidia executive told me that the company had tested the two systems against each other. He said the number of driver takeovers of the Nvidia system was similar, sometimes preferring one system, sometimes the other.

Wu declined to comment directly on Tesla’s safety record, but explained that Nvidia differentiates itself through the use of multiple sensors, including cameras, radar, ultrasonic sensors, and — in higher configurations — lidar. Nvidia believes redundancy and diversity in sensor technologies are critical to handling difficult situations and achieving higher levels of security, Wu said.

Additional sensors mean additional costs. The inclusion of lidar technology, in particular, suggests that Nvidia’s safer system will only be available to wealthy Mercedes owners. But Wu believes Nvidia’s vertically integrated approach allows it to deliver the required safety performance at the lowest possible cost.

Nvidia’s DRIVE Hyperion platform is designed with multiple configurations in mind. The basic version uses a simpler and more cost-effective sensor setup, relying mainly on cameras and radar. These sensors have become significantly cheaper over the past decade due to mass production; Ultrasonic sensors are already very inexpensive. For higher levels of autonomy, the platform can add lidar sensors, and given the low cost of lidar, Wu said he believes vehicles priced in the $40,000 to $50,000 range could realistically include the full sensor suite needed for advanced autonomy.

Advantages and disadvantages of data

I asked Wu about recent safety incidents involving Waymo vehicles, such as the company’s robotaxis Intersections closed during power outages in San Francisco. Nvidia was already running similar edge cases through its own simulators, he said. In fact, the company relies heavily on synthetic driving data to account for its shortcomings in real-world testing. Tesla has billions of real-world miles driven, thanks to its massive fleet of customer cars. Waymo has logged nearly 200 million fully self-driving miles on public roads. How can Nvidia hope to catch up?

“The big infrastructure play is actually a simulation,” Wu said. Nvidia takes two approaches to this. One is neural reconstruction, or NuRec, in which the company’s engineers recreate real-world driving scenarios using sensor data collected from vehicles in the field. The other is augmentation, which adjusts elements within the reconstructed scene to explore different possible outcomes. This allows engineers to explore how the autonomous system behaves under slightly different conditions and identify rare cases that may be present in the original data set.

“We can make pedestrians exit faster and slower in a different location,” he said. “This is what we call blurring the data set.”

Nvidia obtains dashcam footage from its partners to feed it with data it uses in simulations. It also recreates edge cases from these Waymo incidents, such as power outages, and trains its system to respond without blocking intersections.

But the ultimate goal is to build a system that uses logic to avoid these traps, thus avoiding the need for real-world driving data in the first place. Wu’s team is working on what it calls a vision language business model, which will put this theory into practice. These models combine visual perception, language comprehension, and physical action into a unified architecture, relying on large underlying models already trained on Internet-scale datasets. Wu likens him to Ed the driver.

“When we teach a child how to drive, they read the rule book and then practice for 20 hours behind the wheel,” Wu said. “Usually, they’re not bad drivers to begin with – although obviously it takes experience to improve. Ultimately, we want the model to work the same way: in the future, with just a rule book and 20 hours of training data, it will learn how to drive.”