Real-time rendering

Physics

Mathematics

3D rendering

3D computer graphics

Computer graphics

Global illumination

Algorithm

Specular reflection

Optics

Computer graphics (images)

Ray tracing (physics)

Mathematical analysis

Graphics pipeline

Shader

Artificial intelligence

Parameterized complexity

Differentiable function

Rendering (computer graphics)

Inverse

Geometry

Computer science

Computer vision

We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer, and can reconstruct geometry, materials, and illumination from scratch from a set of images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.

PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting