NVIDIA AI Can Transform Simple Doodles into Spectacular Landscapes

NVIDIA’s AI App Can Transform Simple Doodles into Spectacular Landscapes

Researchers at NVIDIA have already developed AI technology that can generate photos of fake people, and now they're taking things a step further. Continuing their exploration of what's possible using generative adversarial networks, or GANs, they created a program that can turn a simple doodle into a beautiful landscape “photograph.”

With a few clicks and drags in an app that looks like MS Paint, anyone can create a scheme that is then translated into a startlingly realistic landscape. NVIDIA's deep learning model takes the basic sketch and uses an algorithm to create a photorealistic masterpiece with ease.

Named GauGAN—a nod to famed post-Impressionist painter Paul Gauguin—the interactive app used to produce the fake landscapes is easy to manipulate. Different colors are assigned to elements like sky, mountain, rock, plant, and road. It's even possible to select weather elements like snow, and the program is smart enough to distinguish between details like gravel and sand.

“It’s like a coloring book picture that describes where a tree is, where the sun is, where the sky is,” explains Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA. “And then the neural network is able to fill in all of the detail and texture, and the reflections, shadows and colors, based on what it has learned about real images.”

This means that if you place a body of water next to trees, the program will know to reflect the trees in the pond in the resulting photograph. You can even show the same landscape in different seasons by toggling between grass and snow. NVIDIA hopes that the program will be useful for people who design virtual worlds, such as architects and landscape designers.

As the program was trained using millions of images, it has a rich knowledge that helps it fine-tune its landscapes. “This technology is not just stitching together pieces of other images, or cutting and pasting textures,” Catanzaro said. “It’s actually synthesizing new images, very similar to how an artist would draw something.”

Interestingly, the underlying technology used in GauGAN can also fill in features like buildings and people, which will surely be a further development.In June, a research paper on GauGAN will be presented at the CVPR conference. In the meantime, attendees of NVIDIA's GPU Technology Conference in March 2019 had a chance to test out these artificial landscapes.