Steered-Mixture-of-Experts (SMoE)

Steered Mixtures-of-Experts (SMoE) is a regression framework for modeling and compression of 2D images, higher-dimensional imagery, including compression of light fields and light-field video, and 3D scenes.
3D SMoE Splatting Project Webpage
As visual computing advances across domains such as image editing, autonomous driving, and digital twins, the need for high-fidelity yet computationally efficient representations has become increasingly critical. Traditional 2D models are constrained by fixed grids, limiting their adaptability and compactness, while emerging 3D techniques often deliver realism at the cost of excessive training time, memory usage, and energy consumption. This PhD project tackles a central challenge across both 2D and 3D domains, and examines Steered Mixture-of-Experts (SMoE)—a modular, kernel-based architecture that promises localized modeling and interpretability.

The three guiding research questions of the PhD project are:

  • How can we design an initialization scheme that yields a compact SMoE model while drastically reducing training time?
  • Can the SMoE architecture itself be restructured to accelerate convergence by an order of magnitude?
  • Can SMoE be extended to represent 3D radiance fields, enabling real-time rendering without trading off reconstruction quality?

Abstract

 

This PhD project investigates an explicit regression model aimed at reducing the number of computational primitives in both 2D image and 3D radiance field domains. The scope of this work includes developing methods to reduce computational primitives, investigating explicit regression for transparency and scalability, and evaluating these methods across 2D image and 3D radiance field domains. This thesis is subject to specific delimitations. First, traditional compression techniques, such as quantization, entropy coding, or transform-based representations, are excluded. These methods are useful for reducing memory either during training or after, but they do not address the structural modeling inefficiencies or change the main architecture. The focus of this thesis is to evaluate the effectiveness of the redesigned architecture. Second, implicit neural representations and deep networks, such as NeRF variants, transformers, diffusion models, or convolutional autoencoders, are not used as baselines or design foundations. These architectures often hide the modeling structure behind layers of nonlinearity and offer limited control over parameter allocation. This thesis instead emphasizes transparent, interpretable models with modular behavior. Third, the thesis works with 3D radiance fields. It omits alternative 3D representations such as point clouds, meshes, ray tracing, or light-field rendering. These modalities introduce separate concerns and would weaken the focus on the architecture and regression method.

 

3D SMoE Splatting for Edge-aware Realtime Radiance Field Rendering

Watch the video on YouTube

We introduce a novel, edge-aware "3D SMoE Splatting" (3DSMoES) framework for 3D rendering is introduced, adopted to fit into the existing "3D Gaussian Splatting" (3DGS) CUDA optimization pipeline. Here, SMoE regression serves as a "plug-and-play" solution that replaces the established 3DGS regression as a novel workhorse. 3DSMoES achieves significant visual quality gains with drastically fewer Gaussian kernels compared to 3DGS.

Publications

List of relevant publications

3D SMoE Splatting for Edge-aware Realtime Radiance Field Rendering

Yi-Hsin Li, Thomas Sikora, Sebastian Knorr, Mårten Sjöström
ACM Siggraph Asia 2025

We introduce a novel, edge-aware "3D SMoE Splatting" (3DSMoES) framework for 3D rendering, adopted to fit into the existing "3D Gaussian Splatting" (3DGS) CUDA optimization pipeline

Adaptive Segmentation-Based Initialization for Steered Mixture of Experts Image Regression

Yi-Hsin Li, Sebastian Knorr, Mårten Sjöström, Thomas Sikora
IEEE Transactions on Multimedia 2025

We introduce a novel adaptive segmentation-based initialization method targeted for optimizing Steered-Mixture-of Experts (SMoE) gating networks and Radial-Basis-Function (RBF) networks with steering kernels.

Segmentation-based Initialization for Steered Mixture of Experts

Yi-Hsin Li, Mårten Sjöström, Sebastian Knorr, Thomas Sikora
IEEE International Conference on Visual Communications and Image Processing (VCIP 2023)

The Steered-Mixture-of-Experts (SMoE) model is an edge-aware kernel representation that has successfully been explored for the compression of images, video, and higher-dimensional data such as light fields. The present work aims to leverage the potential for enhanced compression gains through efficient kernel reduction.

Comments