THIS is the Biggest Thing Since CGI

The Introduction to Holography and 4D Gaussian Splats 00:21

"This is a 4D Gaussian splat. We are witnessing the birth of a new medium."

• Holograms represent a significant advancement in visual media, allowing for real-time 3D reconstructions that can be viewed from multiple angles and even perspectives in time.

• Gaussian splats are at the forefront of this evolution, blending photorealistic representation with real-time rendering capabilities.

• The video sets out to explore this innovative technology, how it functions, and its potential future applications.

Transition from Neural Radiance Fields to Gaussian Splats 00:58

"Nerfs were abandoned because they had been replaced with something better."

• The presenter recalls a previous fascination with neural radiance fields (Nerfs), which allowed for capturing photoreal 3D scenes without extensive work.

• Although initial excitement surrounded Nerfs, they were too slow and cumbersome for industry adoption, prompting a shift to Gaussian splats as a superior alternative.

• Gaussian splats have become increasingly prevalent in visual projects, showcasing their potential through various examples in the video.

Understanding Gaussian Splats and 3D Capture 02:30

"On a fundamental level, there's not really much difference between splats, nerfs, and traditional photo scans because they're each a form of 3D capture."

• Gaussian splats serve as a form of 3D capture that operates similarly to traditional methods, relying on overlapping photographs taken from multiple viewpoints.

• The process of creating a 3D model involves triangulating camera positions and identifying common pixels across images, resulting in a point cloud representation of the scene.

• Unlike traditional photo scanning techniques, Gaussian splats capture view-dependent lighting effects, which is crucial for achieving realism in 3D models.

The Science Behind Gaussian Splats 06:16

"Each Gaussian actually stores a single base color, and the spherical harmonics change that color depending on the viewing angle using a simple math formula."

• Gaussian splats consist of multiple overlapping Gaussians, providing a fuzzy representation that can adapt based on the viewer's perspective.

• These blobs can vary in shape, size, and transparency, enabling them to accurately depict both detailed objects and large surfaces seamlessly.

• The use of spherical harmonics allows for efficient storage and computation of colors, enabling a realistic representation without the need to store every possible shade.

Best Practices for Creating Gaussian Splats 07:17

"You want to have the crispiest images possible. Any amount of blur is your worst enemy."

• To achieve high-quality scans for Gaussian splats, it is critical to capture sharp images with minimal motion blur.

• This requires setting a high shutter speed and stabilizing camera settings such as white balance and ISO.

• The workflow includes capturing data in a structured manner which also supports subsequent tracking for the Gaussian splats to develop in the imaging process.

Gaussian Splat Software Options 08:14

“We have Lightfield Studio, which is free and open-source, but you have to do all your tracking somewhere else.”

• There are various software options available for creating Gaussian splats. These include paid software as well as free, open-source alternatives.

• Lightfield Studio is one such free program, but it requires users to conduct the tracking in a separate application.

• For those familiar with advanced tools, reality scan software and all-in-one packages like Cury Engine or Luma are viable choices for generating these effects.

• The appeal of these software solutions is their ease of use—just import your video, prepare your settings, start the process, and return later to see the results.

Examples of Gaussian Splats in Action 08:39

“This is one of the best Gaussian splats I've ever seen in my life.”

• The video showcases various examples of Gaussian splats from a site named Super Splat, demonstrating how realistic these splats can be.

• One highlighted example was a rendition from a garage, which impressed due to its speed of rendering and high-quality output.

• Details such as spherical harmonics become evident in the splats, as one specific example illustrated a piece of chocolate that changed in color as the viewpoint shifted, mimicking realistic reflection.

• The intricacies of the details, including reflections and textures like cross-hatching in foil, show how Gaussian splats replicate fine details and depth effectively.

High-Frequency Details Captured by Gaussian Splats 10:09

“Gaussian splats excel at high-frequency tiny details like hair and leaves.”

• Gaussian splats are particularly adept at capturing intricate details, such as the hair on animals and the transparency of insect wings.

• The video illustrated an example involving a detailed scan of a dog, where the splat provided depth that would be hard to achieve through conventional photography.

• There was also an impressive display of the splat capturing features of a bee, showing its wing structure and fine details that typically go unnoticed.

Creating Gaussian Splats from Renders 10:39

“You can create a Gaussian splat from any array of images looking at something from different points of view.”

• The process isn’t limited to real-life captures; Gaussian splats can be generated from renders, as demonstrated by a video showcasing a rendered chicken shawarma.

• A significant advantage is that creating a splat allows users to view the render from any angle without the need for complete re-rendering, drastically reducing the time and effort required.

• This technique is also applicable in video games, particularly in "photo mode," where users can pause the game and analyze characters or environments from various angles.

The Capture, Tracking, and Training Process 14:50

“There are three steps to making a Gaussian splat: capture, tracking, and training.”

• The workflow for generating a Gaussian splat involves three essential stages: capturing footage, tracking, and training the software.

• The capture stage involves converting the footage into a format compatible for tracking, highlighting the need for careful consideration during the filming process.

• Tracking poses its own challenges since recording in 360-degree footage requires conversion back into regular formats to ensure proper alignment and mapping.

• Successful training of the model occurs after these steps are completed, which ultimately refines the quality of the generated Gaussian splat.

The Power of Gaussian Splats in Historical Preservation 16:08

“You can literally take your old photo scans and, as long as you have the original data set, you can upgrade them to Gaussian splats.”

• Gaussian splats provide an efficient method for capturing and preserving reality, especially in cases of historical significance.

• By utilizing photo scanning techniques and original data sets, older scans can be reprocessed using Gaussian splats to create more photorealistic representations.

• This technology proves invaluable in historical preservation, allowing for accurate digital records of landmarks before they are lost. For instance, the speaker mentions their effort to reprocess a scan from Inspiration Point, which was destroyed in wildfires, ensuring that a digital version of it remains available.

Enhancing Reality with Visual Effects 17:30

“But I'm a visual effects artist, and my job is to bend reality.”

• Gaussian splats are not just about preservation; they also serve as a foundation for creative enhancements in visual effects.

• Using software like Octane Render, professionals can manipulate scanned environments to create dynamic scenes.

• Techniques like relighting, adding 3D objects that reflect the surroundings, and incorporating particle effects can transform these scans into visually rich imagery. For example, by altering the view of a studio scan and applying various effects, the artist can emulate a miniature macro photography look.

Real-Time Capabilities with Gaussian Splats 19:23

“One of the great things about Gaussian splats is that they run in real time in Unreal Engine.”

• The real-time processing capabilities of Gaussian splats allow for immediate applications in various environments, such as LED walls.

• By tracking the camera in 3D space, virtual cameras can simulate movement in a way that feels natural and immersive. This capability enables the creation of detailed scenes that blend real and virtual elements seamlessly.

• The speaker demonstrates this technology by being at a showroom utilizing 40 Gaussian splats, indicating a significant advancement in realtime visual technology.

The Future of 4D Gaussian Splat Technology 20:58

“To put it simply, it's the hologram that people have been dreaming about in sci-fi movies for decades, but now it's a reality.”

• 4D Gaussian splats represent a leap forward in visual technology, creating immersive holographic experiences.

• This technology enables innovative methods of interaction, allowing users to manipulate visual effects during live interviews and presentations.

• By combining spatial tracking and dynamically flowing Gaussian points, these splats can deliver fluid, high-fidelity visuals with minimal latency. This opens new doors for creative applications, transforming how content is produced and consumed.

Key Advantages of Continuous Gaussian Representation 23:12

“Each Gaussian point in our representation has a velocity and a time span.”

• The innovative technique of continuous representation means that a single Gaussian point can encapsulate multiple frames of information.

• This allows the system to compress extensive data into manageable sizes without compromising quality.

• The ability to achieve high frame rates, such as 1,000 FPS or more, while maintaining visual integrity is a defining feature of this technology. By reducing data sizes to levels that can be easily processed by devices, like smartphones or VR headsets, 4D Gaussian splats democratize access to advanced visual experiences.

The Magic of foring and Holograms 08:00

"That's the magic of foring."

• The concept of "foring" refers to an advanced technology that creates immersive experiences, allowing people to feel as if they are in different realities or locations.

• This technology, likened to scenes from popular shows such as "Stranger Things," enables users to perceive themselves in virtually any environment, enhancing the storytelling aspect of filmmaking.

The Future of Gaussian Splat Technology 08:51

"I think we're going to start seeing splats rolled out in all kinds of places."

• Gaussian splats are poised to be integrated into various applications, like Google Maps, replacing traditional views with more immersive formats.

• Companies like Zillow are also introducing Gaussian splats in real estate listings, allowing potential buyers to view properties from multiple perspectives in a highly realistic manner.

Potential and Challenges of 4D Gaussian Splat Technology 09:18

"Splats are great right now because they're easy to use, but they're not easy to make."

• The production of 4D Gaussian splats requires a significant number of cameras and substantial computational power to create and refine these models.

• Current technological limitations present challenges that are expected to be resolved in the near future, potentially easing the process of creating realistic splats for broader applications.

The Evolving Landscape of Holograms 09:39

"If we consider radiance fields and all of that to be a form of hologram, I think holograms are here to stay."

• The notion of holograms is expanding with advancements in technology, indicating that they're becoming a staple in media alongside pictures, videos, and sound.

• The anticipation of widespread adoption of holograms suggests they will play a significant role in the future of virtual reality and digital interactions.