Introduction
Gaussian Splatting is a powerful technique that allows for detailed compositing in 3D scenes, especially when integrating real-world footage with virtual elements. In this tutorial, we will explore the process of tracking footage shot on an iPhone or another camera, using Meta Shape for data transformation, and bringing this into After Effects to achieve precise compositing. The steps outlined will help you leverage the capabilities of Gaussian Splatting to enhance your visual effects workflow.
Tracking and Preparing Your Footage
To start, import your footage into SynthEyes, a popular tracking software. This tutorial does not delve deeply into the tracking process, as it focuses on the broader workflow. However, we will cover the essential steps to get you started.
Step-by-Step Tracking in SynthEyes
1. Import Footage: Load your frames into SynthEyes and set the correct frame rate.
2. Auto Tracking: Use the auto-tracker to place default tracking points on your footage.
3. Refine Tracking Points: Adjust settings in the features panel to add more tracking points and ensure accurate data.
4. Manual Adjustments: Delete any incorrect tracking points, such as those on moving objects or unstable surfaces.
5. Solve the Scene: Use the solver panel for an automatic solve. Refine the tracking data until the error is below one pixel for better accuracy.
Exporting Data for Meta Shape
Once your tracking data is ready, you need to export it for use in Meta Shape (formerly known as Agisoft PhotoScan).
Export from SynthEyes
1. For After Effects: Export the tracking data as a JavaScript file for After Effects.
2. For Meta Shape: Export the data as an XML file to be imported into Meta Shape.
Using Meta Shape for Point Cloud Generation
Meta Shape helps generate a point cloud from your tracked footage, which is crucial for the Gaussian Splatting process.
Steps in Meta Shape
1. Import Image Sequence: Load your tracked frames into Meta Shape.
2. Import Cameras: Import the XML file exported from SynthEyes.
3. Generate Point Cloud: Run the process to generate a dense point cloud from your footage.
4. Clean Up: Optionally, clean up the point cloud by removing floating points that don’t contribute to the final composite.
Training the Gaussian Splat
With your point cloud ready, the next step is to train the Gaussian Splat file using Post Shot.
Post Shot Workflow
1. Import Point Cloud: Drag the point cloud file into Post Shot and disable down-sampling for high-quality results.
2. Training: Allow Post Shot to process the data, which may require a powerful GPU. Delete unnecessary points early in the training to save time.
3. Export PY File: Once trained, export the data as a .py file, which will be used in After Effects.
Compositing in After Effects
Finally, bring your trained Gaussian Splat file into After Effects to integrate it with your original footage.
Steps in After Effects
1. Import Files: Import your original footage and the .py file into After Effects.
2. Setup Composition: Create a new composition and drop in your footage and a solid layer.
3. Apply Gaussian Splat Tool: Use the appropriate tool, such as Potion Splatting from ExrX, to apply the Gaussian Splat to your solid layer.
4. Align Data Sets: Ensure the Gaussian Splat aligns perfectly with your original footage by adjusting the axes if necessary.
5. Composite Elements: Use the Gaussian Splat to place objects behind fine details, enabling complex compositing that would be challenging with traditional roto techniques.
Conclusion
This tutorial demonstrates how Gaussian Splatting can significantly enhance your compositing workflow by allowing precise placement of virtual objects within real-world footage. By following these steps, you can achieve seamless integration of detailed elements into your scenes, opening up new possibilities for creative visual effects. Whether you are working on professional projects or personal experiments, mastering Gaussian Splatting will elevate the quality of your composites.