To create a digital painting, we need to develop a generative model first. Now, what is meant by a generative model? In the context of generating images, they can be thought of as a machine learning model that learns the probability distribution of every combination of pixel values. If we can develop a model that learns this distribution, sampling in that distribution, the combination of pixels that have the larger probability, we can generate new images that share features to the images we are trying to model.
One such generative model is called generative adversarial networks (GAN). Yann LeCun who was the shared recipient of the 2018 Turing Award described GAN’s when it was first published to be the most interesting idea in the last 10 years in Machine learning.
A GAN contains two parts mainly
1) Generator: Learns to generate fake images to fool the discriminator. They can be thought of as s forger who tries to make counterfeit money.
2) Discriminator: Learns to distinguish whether the images seen by it is real or fake. They are similar to a detective that tries to catch counterfeit money.
The typical steps involved are the following:
1) The generator network creates a fake image from a random distribution
2) These fake images are passed to the discriminator as part of a series of other images taken from the actual/true images.
3) The discriminator looks at each of these real/fake images and makes the prediction whether the image it sees comes from real dataset or from the generator
4) The generator then gets feedback from the discriminator on how good the fake images generation is and generator updates itself accordingly
5) Similarly, the discriminator gets updates based on how well it classifies fake from real images.
These two networks compete with one another to improve. At the start of training, the generator produces a random noise image. With training, the generator received feedback from the discriminator network, which helps create better and realistic images. The discriminator also improves with time in its ability to detect fake from real images making it harder for the generator to trick it. The generator's goal is to produce examples that capture the characteristics of the images we are trying to model and should be indistinguishable from the training data.
In our study, we use a GAN architecture called StyleGAN to model the artworks of the artist. A total of 160 images of the artist’s prior work were initially randomly cropped to multiple 512 px resolution images to increase the sample size to a total of 16,000 for training the generative model. Instead of generating the images from Gaussian noise typically done in GANs, we condition it on the artist’s brain activity acquired with scalp electroencephalography (EEG). The minimally denoised EEG was mapped into a latent space using the mapping network which was then used to train the generator. The following are some of the images created by our model on the right sampled randomly. The figure on the left corresponds to the real images that had the close resemblances to the generated paintings.
The evolution of the training is shown below. At the start, the images were highly random but towards the end, the GAN started to the learn pattern and the colors schemes used by the artist
