Real-Time Light Field Reconstruction and Neural Rendering

We introduce a lightweight, position-aware network designed for real-time view synthesis from a single input image and a target camera pose. The proposed framework consists of a Position Aware Embedding, which efficiently maps positional information from the target pose to generate high dimensional feature maps. These feature maps, along with the input image, are fed into a Rendering Network that merges features from dual encoder branches to resolve both high and low level details, producing a realistic new view of the scene.

We propose a unified multimodal network capable of jointly synthesizing new views and predicting consistent depth maps from a single input image. Our framework integrates an additional depth prediction module into a state-ofthe-art view synthesis architecture by leveraging a shared latent representation, thereby ensuring geometric coherence between synthesized views and their depth maps.

We propose a fully learning-based method for spherical light field reconstruction from a single omnidirectional image. The proposed LFSphereNet utilizes two different networks: The first network learns to reconstruct the light field in cubemap projection (CMP) format given the six cube faces of an omnidirectional image and the corresponding cube face positions as input. The cubemap format implies a linear re-projection, which is more appropriate for a neural network. The second network refines the reconstructed cubemaps in equirectangular projection (ERP) format by removing cubemap border artifacts.