Instafamous with Robo-Bob Ross
Social ‘influencers’ are a relatively new societal phenomenon in which members with many connections on a web site are capable of delivering content to the eyes of many people. For users with such a large collection of followers comes great power and, more importantly, great potential for profit. Headlines advertising the oppurtunities are plenty:
Entry-level users with 1,000 followers can earn £4,160 per year if they just post twice a week
…with around 100,000 followers, the going rate is $5,000 per post.
Top Instagram Users Making Thousaunds Per Photo by Promoting Products
…$100-$250 per post if you have approximately 50,000 followers
Accounts with 17,000 follewers can earn £26,520 a year
With such lucrative business ventures looming, it seems imperative for us to partake! So what does it take to gain such a massive and profitable following? Let’s take a look at the Instagram members with the most followers and try to gain some insight:
🤔 The top Instagramers are all, perhaps unsurprisingly, those with large, pre-established, fanbases: celebrities. Although something we can dream of (100 million followers must mean a lot of $$$), the chances of becoming an over-night sensation and immediatly having a huge fan base to funnel into my new Instagram venture is slim.
Looking to some lesser profiles may be yeild something a little more actionable. Instagram being a visual platform means those who are skilled at visual art should find they have a leg up and, in fact, there are quite a few b-list celebrity artists and hobyists alike that do well quite well. Take a look at some examples:
If the headlines are to believed, tens to hundreds of thousands of followers could provide me with a nice chunk of pocket change. But there’s a problem: I’m no skilled artist. Though maybe I could learn, after all, one of the most renowned teachers of all time is an artist: Bob Ross.
Unfortunately, there are hundreds of Bob Ross episodes, each about 30 minutes. It may be weeks of painting along until there’s some semblence of a Bob Ross quality painting ready for recieving those precious Instagram hearts, not to mention the cost of paint. I may be thousands of dollars in the hole before I get a single sponsor! The only choice left is: clone Bob Ross into a computer.
The Birth of Robo-Bob Ross and his Quest for Instafame
How’s this going to work, then? Fortunately, there’s been much hype around GANs, or Generative Adversarial Networks, being used to create new images in the style of images they have been trained on given some form of input. In this case, should be able to train a GAN on Bob Ross’ collection of paintings, then taking some input images chosen from trending images in select hashtags on Instagram, then running them back through the GAN to produce some quality content ready for posting. However, the first iterations may not be of that high quality, so we may plan to tune parameters by hand or proportionally to the number of likes a post recieves through some reinforcement technique. More succinctly, the process may be illustated as below:
Developing a content candidate filter
A lot of photos retrieved from a specific hashtag, eg: #landscape, aren’t very representative of a landscape, or may simply not be worth trying to stylize. To filter out the obvious bad content candidates, we could train a new image classifier on some labeled data, however, training a deep network like this is unfeasable on my laptop or in any reasonable time. Another, common approach, would be to utilize a pre-trained network, such as one from the ILSVRC, and train an additional layer on the network on my specific data set. This transfer learning approach will significantly speed up the learining rate, however, it will still involve training a rather large network on my laptop.
The final choice that I settled with is a more confined version of the transfer learning approach. I will utilize a pretrained image classifier to produce classification tags for an image, then train a much simpler model on those tags to produce predictions.
Hand labeling a data set yourself may sound rather off-putting to many, but not to fear, it’s actually quite easy in this case. I started with only about 500 images from the #landscape hashtag. To label them, I merely used finder (or any file browser) to select all the images I knew would not work well, then move them into a rejects directory. The rest went into an accepted directory. The process only took about 15 minutes and the results were quite good!
257 Bad landscapes 308 Good landscapes
Producing Training Data (meta-features)
Seeing as I’m trying to avoid traning any sort of neural network, I need to extract features out of the images. I produce these meta-features using the pretrained VGG19 model trained on ImageNet for the ILSVRC-2014. Keras makes this very easy by building it into a utility library.
I then run this model over all the training images and generate tags (meta-features) for each one and store them all in a .csv.
Learning to Paint
Neural style transfer was introduced a
couple years ago as method of applying one image’s artistic style to another
image. Style transfer works by training a CNN on a
content image and another
CNN on a
style image. It then swaps out one convolution layer in the
then reconstructs an image from the network. This results in an image in the
same artistic style and color of the
style image, but with the subject of
content image and tends to work quite well (see paper). However, after
experimenting a lot with
neural-style-tf and it’s handful
of paramaters, I found that the results were too inconsistent and often filled
with artifacts. Here’s an example:
This looks akin to a Photoshop filter, only worse. Applying rule based filters would likely be much better and much less CPU intensive than trying this approach with GANs.
Instead of trying to adapt neural transfer, I instead opted to try CycleGAN. CycleGAN is able to translate any image from one space (photo) to another (Bob Ross painting) using only images from each category (they don’t even need to be paired). It does this by contstructing two GANs for each category: one to translate images from category A to category B, the other for images from category B to category A. Usinig CycleGAN, I can generate a Bob Ross painting not by simply copying a single image’s style to another, but applying a learned knowledge of many Bob Ross paintings to an image!
This approach, however, required training a model (two GANs) on a set of Bob Ross paintings and another set of landscape photos. It turned out the CycleGAN source included a repository of over 6000 landscape photos from flikr and I used The Bob Ross Painting Index to grab about 400 of Bob Ross’ paintings (screen captures taken from his show). The training was very slow. At first, I only could utilize a cpu which was only giving me about one epoch per day (!!). I only was able to bear it up through about 5 epochs before I had a GPU I could put to the task. I was able to get some nice results after about 50 epochs, or two days, of training.
Below are shown some ‘cycles’ after various amounts of training epochs where a cycle is a Bob Ross painting after being transformed into a photo, then from that, transformed back to the Bob Ross painting.
Try it Out
At this point, I have a GAN that has been trained as much as my patience will allow. It’s time to try it out on some #landscape photos:
These are a random sampling of painting generated from my collection of #landscape posts that I have downloaded. It may also be interesting to see the original photo next to the result painting to see how much artistic freedom Robotic Bob Ross is taking.
There’s only one problem with CycleGAN: I’m unable to generate pictures much larger than 256x256. My GPU only has 4GB of ram, which limits the size of the generator network I’m able to load. It also seems Torch models trained on GPU also won’t work on CPU, although I need to investigate further. Instagram scales everything to 1080x1080, which would make my images look rather rediculous, so rather than trying to scale up a single image, I’ll combine a couple to make photo sheets. This is less than optimal because it won’t allow me to record feedback image by image, but there’s not much better options at the moment.
Hashtags are a central part of Intsagram and play a large part in determining the success of a post and the growth of a userbase. There’s a whole problem of optimizing the hashtags to reach the largest user base possible, but for now, remember the tags we had VGG generate for filtering out bad photos? We can use those as hashtags!
It’s nice to then visualize what a candidate post looks like before going and posting it to Instagram. This gives a point where human intervention can take a place to select out the best candidate submission for the day and creates an oppurtunity to train some reinforcement model to learn how to do this later.
Now that we’ve reviewed the possible posts for the day, we can go ahead and upload the best one to my soon to be famous Instagram profile! Here’s a couple examples in the final form:
Most of these hashtags have nothing to do with the paintings, as far as I can tell, but maybe the GAN knows something I don’t…
At this point, I’ve successfully automated the difficult and expensive task of learning how to paint decnently, as well as any time that would be required to actually produce subsequent paintings. There is of course lots of other things to automate here, such as generating posts daily and uploading the best automatically, but these tasks are relatively straight foreword in comparison.
References and Further Reading
- Image Style Transfer Using Convolutional Neural Networks