SceneDreamer is a generative model that can create vast 3D landscapes from random noises without any 3D annotations. It is a powerful tool that can synthesize large-scale 3D environments from in-the-wild 2D image collections. At its core, SceneDreamer employs a novel bird's-eye-view (BEV) representation of 3D scenes, which consists of a height field and a semantic field. The height field represents the surface elevation of 3D scenes, while the semantic field provides detailed scene semantics. The BEV scene representation enables SceneDreamer to represent a 3D scene with quadratic complexity, disentangle geometry and semantics, and perform efficient training.
To parameterize the latent space of the model, SceneDreamer proposes a generative neural hash grid that encodes generalizable features across scenes and aligns content. This approach allows the tool to generate vivid and diverse unbounded 3D worlds that are superior to state-of-the-art methods in this regard. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. This renderer enables SceneDreamer to generate highly realistic images of the 3D environments it creates.
SceneDreamer's ability to generate vast and diverse 3D landscapes from random noises makes it a powerful tool for a wide range of applications, including video games, virtual reality, and architectural visualization. Its ability to disentangle geometry and semantics also makes it a valuable tool for 3D scene understanding and generation. With its efficient and expressive representation, SceneDreamer has the potential to revolutionize the field of 3D scene generation and representation.
