3D Gaussian Splatting, described in 3D Gaussian Splatting for Real-Time Radiance Field Rendering, is a rasterization technique that enables real-time rendering of photorealistic scenes learned from small samples of images. This article details how it works and what it means for the future of graphics.
What is 3D Gaussian Splatting?
3D Gaussian splatting is essentially a rasterization technique. In other words:
Prepare data that describes the scene. Draw the data on the screen.
This is similar to triangle rasterization in computer graphics, which is used to draw large numbers of triangles on the screen.
However, it is Gaussian, not a triangle. This is a single rasterized Gaussian distribution with borders drawn for clarity.
This is explained by the following parameters.
Position: Where it is placed (XYZ) Covariance: How it is stretched/scaled (3×3 matrix) Color: What color (RGB) Alpha: Transparency (α)
In fact, multiple Gaussians are drawn at once.
That’s 3 Gauss. So what about 7 million Gauss?
If we rasterize each Gaussian to be fully opaque, it will look like this:
This is a very brief overview of what 3D Gaussian splatting is. Now let’s take a look at the complete steps mentioned in the paper.
structure
1. Structure seen from movement
The first step is to estimate a point cloud from a set of images using a Structure from Motion (SfM) method. This is a method for estimating a 3D point cloud from a set of 2D images. This can be done using the COLMAP library.
2. Conversion to Gaussian
Each point is then converted to a Gaussian. This is sufficient for rasterization. However, only position and color can be estimated from SfM data. To learn a representation that produces high-quality results, you need to train it.
3. Training
The training procedure uses stochastic gradient descent, similar to neural networks, but without layers. The training steps are as follows:
Rasterize the Gaussian to the image using differentiable Gaussian rasterization (more on that later) Compute the loss based on the difference between the rasterized image and the ground truth image Adjust the Gaussian parameters depending on the loss Apply automatic densification and pruning
Steps 1-3 are conceptually very simple. Step 4 includes:
If the given Gaussian gradient is large (i.e. too wrong), split/clone. If the Gaussian is small, clone it. If the Gaussian is large, divide it. If the Gaussian’s alpha is too low, remove the Gaussian.
This step allows you to fit the Gaussian more closely while removing unnecessary Gaussians.
4. Differentiable Gaussian Rasterization
As mentioned earlier, 3D Gaussian splatting is a rasterization approach that draws data to the screen. However, some important factors are:
Fast differentiable
The original implementation of the rasterizer can be found here. Rasterization includes:
Project each Gaussian into 2D from the camera’s perspective. Sort the Gaussian distribution by depth. For each pixel, repeat each Gaussian from front to back and blend them.
Additional optimizations are discussed in the paper.
It is also important that the rasterizer is differentiable so that it can be trained with stochastic gradient descent. However, this only concerns training, and a trained Gaussian can also be rendered with a non-differentiable approach.
who cares?
Why is 3D Gaussian splatting gaining so much attention? The obvious answer is that the results speak for themselves. That means high quality scenes in real time. However, there may be more to this story.
There are many unknowns about what else Gaussian splatting can do. Can it be animated? The following paper, Dynamic 3D Gaussians: tracking by Persistent Dynamic View Synthesis, suggests that it is possible. There are many other unknowns. Will they be able to reflect? Can I model without training on reference images?
Finally, there is growing research interest in embodied AI. This is an area of AI research where state-of-the-art performance is still orders of magnitude lower than human performance, and many of the challenges lie in the representation of 3D space. Given that 3D Gaussian splatting provides a very dense representation of 3D space, what are the implications for embodied AI research?
These questions call attention to the method. It remains to be seen what the actual impact will be.
The future of graphics
So what does this mean for the future of graphics? Well, let’s break it down into pros/cons:
Strong Points
High-quality, photorealistic scenes Fast real-time rasterization Relatively fast training
Cons
High VRAM usage (4GB for display, 12GB for training) Large disk size (>1GB per scene) Not compatible with existing rendering pipelines Static (currently)
So far, the original CUDA implementation has not been adapted to production rendering pipelines such as Vulkan, DirectX, or WebGPU, so it remains to be seen what the impact will be.
The following adaptations have already been made:
Remote Viewer WebGPU Viewer WebGL Viewer Unity Viewer Optimized WebGL Viewer
These rely on either remote streaming (1) or traditional quad-based rasterization approaches (2-5). The quad-based approach is compatible with decades of graphics technology, but may result in lower quality and performance. However, viewer #5 shows that despite the quad-based approach, the optimization tricks can yield higher quality and performance.
So will 3D Gaussian splatting be completely reimplemented in production? The answer is probably yes. The main bottleneck is the multi-million Gaussian sort. This is efficiently performed in the original implementation using CUB device radix sort, a highly optimized sort available only in CUDA. However, with enough effort, it is certainly possible to achieve this level of performance with other rendering pipelines.
If you have any questions or would like to get involved, please join the Hugging Face Discord.
