A group of researchers from Google DeepMind has published their recent work at the intersection of deep reinforcement learning (Deep RL) and Generative Adversarial Networks training. In their paper named “Unsupervised Doodling and Painting with Improved SPIRAL”, researchers investigate the use of RL agents as generative models for synthetic image generation.
The project, in fact, explores the possibility of training reinforcement learning agents to learn to paint realistic portraits within a GAN framework. For this purpose, researchers designed and built a specific painting environment where a “generator” agent will try to paint portraits whose realism will be judged by a discriminator network in a Generative Adversarial Training setting.
The generator network or the “painting RL agent” was built on top of the SPIRAL architecture proposed by Ganin et al., in 2018. Researchers slightly modified this architecture to fit the purposes of the project and proposed SPIRAL++ as a novel improved generator model.
The SPIRAL approach proposed by Ganin et al., had the objective to train a policy π that controls a black-box rendering simulator R using adversarial learning. At every time step, π observes the current state of the drawing and produces a rendering command used by the rendering simulator to update the canvas. A discriminator network is then used to provide the reward signal for training the generator agent to produce more realistic images. In this work, four improvements to the original SPIRAL architecture were proposed: spectral normalization, temporal credit assignment, a complement discriminator and a population discriminator.
With these improvements, researchers were able to prove that when sufficiently constrained, generative agents can learn to produce images with a degree of visual abstraction while maintaining a high degree of realism. They showed that this kind of framework combining RL agents in a Generative Adversarial Training setting, can have numerous potential applications in the creative domain.
More details about the improved architecture SPIRAL++ and the results obtained can be read in the official paper or on the project’s website.