Using Gaussian Splats to Simulate Digital Painting Brush Strokes

From Pixelation to Painting

The project originated from an earlier edge-aware pixelation tool that deformed a grid to follow image edges for pixel art. This technique effectively preserved details and kept edges crisp. The developer realized that the same edge information, which marks boundaries and contours, could be applied to guide brush strokes in a digital painting context.

Computational Brush Strokes with Gaussians

In computational terms, a brush stroke was conceptualized as an elongated mark with a fading color center and specific orientation. A 2D Gaussian splat was found to map well to this concept. It includes a mean for stroke placement, a covariance matrix for stretching and rotation, and properties for color and opacity. The covariance matrix specifically encodes the brush's direction and elongation, with its major axis aligning with the stroke and minor axis across it.

Simulating Natural Painting Effects

Rendering a field of these Gaussian splats using standard over-compositing, where each splat occludes what is behind it based on its alpha, produces an effect similar to natural painting. This method allows for broad blocks of color to be laid down first, with finer details built on top using smaller, more translucent marks, mimicking the layering process of traditional painting.

Interactive Tool Development

The goal was to create an interactive tool where users could adjust sliders and see the painting reform in real-time. This project also served as an opportunity to evaluate the Jolt framework for building complex applications.