Current technology allows users to create photorealistic images from simple sketches. A recent study on arXiv.org proposes to extend neural rendering to semantically labeled 3D block worlds. The suggested method converts a camera trajectory in Minecraft to a sequence of view-consistent images.
As there is no paired 3D and ground truth real image data, the model is trained on pseudo-ground truth photorealistic images. A novel neural rendering architecture that uses adversarial losses is also proposed. The reconstruction loss lets the user control scene semantics and output style image with a style image. The method efficiently represents large and complex scenes and outperforms current baselines. It can also be used in other 3D block world representations besides Minecraft, such as voxels. These abilities can be used in a wide range of applications, like rapid prototyping for artists.
We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at this https URL .
Research paper: Hao, Z., Mallya, A., Belongie, S., and Liu, M.-Y., “GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds”, 2021. Link: https://arxiv.org/abs/2104.07659v1