The technology needed to build a holodeck

When I visited my elderly mother in Germany recently, I realized it might be one of the last times I would see her in the cozy little house she has called home for more than two decades. So I did what anyone would do: I took out my phone and took lots of pictures of the place to preserve as many memories as I could: the cozy fireplace; The shelves are full of familiar books; The old, rickety park bench that everyone fell upon during a special birthday celebration many years ago.

Then I tried something else. I opened scaniverse3D scanner application from Pokemon Go maker Niantic, and I captured some of these as 3D objects, crouching and tiptoeing my way around them while slowly moving my phone to record every angle and inch. The results were a little patchy around the edges, but it still looked deep. When I opened the scans later, either on my phone or with a VR headset, I was able to look at the worn-out park bench from all angles, as if I were standing directly in front of it. This experience affected me emotionally in ways I was not prepared for.

This experiment was possible thanks to Gaussian splatting, a new method of capturing 3D images that was invented less than two years ago and that has already revolutionized the technology industry. Both Niantic and Google use it to create their own mapping products; Snap has added support for smudges — colloquially called objects captured using Gaussian scattering — to its Lens Studio developer platform, and Meta wants to use Gaussian smudges to create a metaverse that looks just like the real world.

Technology companies are fascinated by Gaussian spraying for its ability to realistically capture 3D objects and then recreate them digitally. It could soon allow anyone to scan entire rooms and change how creatives in Hollywood and beyond record 3D video. When combined with generative AI, it has the potential to not only preserve existing spaces but also transport us into entirely new 3D worlds.

“It’s a big game-changer,” said AR/VR expert and investor Thipatat Chinnavasin. As co-founder and general partner of Reality Investment FundChinavasin has a financial interest in the success of this technology. As a geek and former 3D artist, he fell in love with it, likening it to Star Trek Holodeck, which allowed crew members to enter 3D simulations of real and imaginary spaces. “We’re starting to get to true 3D.”

Capturing things in 3D, even on your phone, is nothing new. However, most previous efforts have relied on polygons, the kind of cybernetic-looking triangular meshes you’ve seen if you’ve ever used an augmented reality app on a mobile phone.

Polygon mesh-based 3D capture and reconstruction is good enough for basic objects with flat surfaces, but may struggle to handle detailed textures and complex lighting. Objects captured this way often look plasticky and unreal, and humans captured in 3D always look like they used too much gel rather than getting individual hair strands. “It was promising at the time, but it always had significant limitations,” Chinnavasin said.

That all changed in the summer of 2023, when a group of European scientists published a paper on something they called “3D Gaussian splatting.” Their approach to the problem was to do away with meshes and instead capture 3D objects as a collection of fuzzy and transparent blobs, also known as Gaussians.

Each one of these blobs is captured with precise information about its color, location, size, rotation, and level of transparency — and when you add up millions of them, you get a more detailed picture of a 3D object that also details what it looks like from any given angle, thanks to all that extra data. Using machine learning, they were able to capture objects in more detail and at a higher resolution and display them in real time without the need for heavy graphics display hardware.

Experts in the field were immediately amazed by the results. “We finally have the opportunity to get real 3D images,” Chinnavasin said. “It’s JPEG’s moment for spatial computing.”

Brian McLendon, senior vice president of engineering at Niantic, believes Gaussian graphics are the most profound advance in 3D graphics in more than 30 years. “We consider it a fundamental change,” he said.

According to McLendon, Gaussian spraying will democratize 3D capture — and Niantic wants to be at the forefront of this change. After obtaining the Scaniverse application In 2021Niantic has added Gaussian spraying as a capture technology last year. In August, it was launched New version From Scaniverse which puts spraying front and center. In October the company Open source Its file format for patches. In December, the Scaniverse service expanded into virtual reality, allowing users to look at Gaussian spots In Meta Quest headphones.

Niantic has its own reasons for pushing spraying. Scaniverse started out as an app for picking up personal mementos and other odd items, but Niantic now encourages people to scan statues, fountains and other public points of interest as well. The company views these inspections as key components of A 3D map of the world you are building – The same map that powers Pokemon Go, PeridotAnd future geospatial augmented reality games and experiences. “We’re very focused on the map, scanning and rebuilding the outer areas,” McClendon said.

“We literally have hundreds of thousands of these (types of scans) in Scaniverse now,” McClendon said. “We hope to reach one million soon.”

Gaussian patches aren’t just for capturing static content. Start computer vision Grace I The technology is used to record 3D videos, which can be viewed on Meta Quest headsets. One of those clips The chef appears Preparing a meal, with viewers able to look at the action from all angles in VR and even zoom in to watch his knife carve off a glistening piece of raw salmon.

Gracia recorded this video in a professional 3D studio, using a set of 40 cameras pointed at the chef from all angles. This is how professionals have been recording 3D content for AR and VR experiences for years – but again, moving from polygons to Gaussian patches makes a big difference.

Previously, 3D video capture presented a series of visual challenges that led to a strict dress code for captured individuals: no crowded patterns, nothing transparent, and nothing loose or dangling that could lead to strange artifacts. When Microsoft caught David Attenborough in this way several years ago, He even had to tape his collar to his shirt And use generous amounts of hairspray to avoid any loose ends that can ruin the shoot.

With Gaussian patches, all these limitations disappear. “There are no restrictions on clothing, there are no restrictions on hair,” said Georgiy Vysotsky, co-founder and CEO of Gracia, who counts Chennavasin’s Venture Reality Fund among his company’s investors. While the previous generation of volumetric video capture required massive amounts of light to remove any shadows, the Gracia was able to record scenes in almost total darkness. “You can leave all the shadows and use artistic lighting,” Vysotsky said. “It’s amazing how much creative flexibility you get with Gaussian patches.”

This does not mean that there are still challenges. Currently, Gaussian clips still require 9GB of data per minute of video — a very large amount for streaming or anything beyond a short technical demo. The company is now working to reduce it to 2-3 gigabytes per minute, and 180-degree VR videos can require less than 1 gigabyte of data per minute, Vysotsky said. He envisions these types of clips eventually replacing instructor recordings in virtual reality workout apps like Supernatural or professional educational content because they allow users to look at instruction from all angles.

One of the most ambitious demonstrations of Gaussian patches to date has been created by Meta. Hyperscapewhich the company unveiled at the Meta Connect conference this fall, is an application for Meta’s Quest headsets that allows users to explore realistic 3D renders. The app launched with six scanned spaces, including five artist studios and a conference room on the Meta campus that served as Mark Zuckerberg’s office.

Hyperscape lets you move freely in these spaces, which is a great experience with that kind of visual fidelity. You can browse through many curiosities at mixed-media artist Diane Hoffman’s San Francisco studio, which includes countless dolls and a “snake skin and shells” box. You can marvel at visual artist Daniel Arsham’s extensive Porsche collection, and even catch a glimpse of the ferns and trees outside the window of Zuck’s former office. The displays look so real that Meta felt compelled to include a warning not to lean on any of the furniture pictured.

For now, Hyperscape is nothing more than a dedicated tech demo. However, Meta has big plans for Gaussian patches, Meta Horizon OS and Quest VP Mark Rabkin told me at Meta Connect this fall. “Gaussian oscillations actually work for us on an engine that looks very similar to the Horizon engine,” Rabkin said, referring to Meta’s social virtual reality platform. “So the technology path to getting it to work in a world is very short.”

Meta envisions it as another tool for VR creators to build immersive worlds and experiences Horizon worlds. The company also has plans to eventually let anyone scan their home and then upload a digital copy of it to the metaverse. “Absolutely,” Rabkin said. “This is what we work for.”

It is unclear how long this work will take, and whether Horizon worlds Whether it will remain in its current form until then is another question entirely. Meta declined to participate in follow-up interviews for this story, but Niantic’s McClendon cautioned not to underestimate the complexity of building a scanning tool like Hyperscape.

“They basically produced a perfect vision,” McClendon said. He suggested that the Meta likely combined multiple scans for each room and may also have done a significant amount of manual editing and cleaning. Since the resulting scans are too large to process in real time on the device, Meta displays them in the cloud and streams them directly to the headset.

“It’s not a metric, but it looks really good,” McLendon said. “Do they have a way to expand this? I don’t know.”

The development of Gaussian sputtering technology is progressing at a rapid pace. McLendon told me that the speed with which new scientific research is emerging on this topic mirrors the speed of generative AI research. “The papers are being published very quickly now,” he added. “The excitement is real.” The technology they are developing is being implemented quickly, Chennavasin said. “Or turned into startups.”

One area ripe for a breakthrough is the combination of strikes and artificial intelligence. Generative AI could improve the capture and display of Gaussian specks, which could allow a company like Gracia AI to capture videos using far fewer cameras. At the same time, more people capturing 3D objects and scenes will greatly increase the amount of high-quality training data for generative 3D video models.

All of this points to a future where ordinary people will be able to create realistic 3D spaces using AI prompts, blurry shots, or a combination of both, and then enter those spaces using virtual reality headsets or augmented reality glasses.

“XR’s killer app is 3D multiplayer,” Chinnavasin said. “Generative AI and Gaussian speckles are how we create them with visual accuracy that is almost indistinguishable from reality. This doesn’t happen overnight. But it’s a clear snapshot now.”

Such a future at hand raises the question: If you had a holodeck, what would you visit first? Reality shows for faraway places you haven’t had the chance to travel to yet? Famous recording studios, museums and libraries? Or rather, fantastic worlds like medieval castles, dungeons or Marvel movie sets?

For me, it might just be my mom’s cozy little house and a rickety park bench.