Complete code examples for Machine Translation with Attention, Image Captioning, Text Generation, and DCGAN implemented with tf.keras and eager execution

By Yash Katariya

I’ve always found generative and sequence models fascinating: they ask a different flavor of question than we usually encounter when we first begin studying machine learning. When I first started studying ML, I learned (as many of us do) about classification and regression. These help us ask and answer questions like:

  • Is this a picture of a cat or a dog? (Classification)
  • What’s the probability that it will rain tomorrow? (Regression)

Classification and regression are incredibly useful skills to master, and there’s nearly no limit to the applications of these areas to useful, real-world problems. But, there are other types of questions we might ask, that feel very different.

  • Can we generate a poem? (Text generation)
  • Can we generate a photo of a cat? (GANs)
  • Can we translate a sentence from one language to another? (NMT)
  • Can we generate a caption for an image? (Image captioning)

During my summer internship, I developed examples for these using two of TensorFlow’s latest APIs: tf.keras, and eager execution, and I’ve shared them all below. I hope you find them useful, and fun!

  • Eager execution is an imperative, define-by-run interface where operations are executed immediately as they are called from Python. This makes it easier to get started with TensorFlow, and can make research and development more intuitive.
  • tf.keras is a high-level API for defining models with lego-like building blocks. I implemented these examples using Model subclassing, which allows one to make fully-customizable models by subclassing tf.keras.Model and defining your own forward pass. Model subclassing is particularly useful when eager execution is enabled since the forward pass can be written imperatively.

If you’re new to these APIs, you can learn more about them by exploring the sequence of notebooks on tensorflow.org/tutorials, which contains recently updated examples.

Each of the examples below is end-to-end, and follows a similar pattern:

  1. Automatically download the training data.
  2. Preprocess the training data, and create a tf.data dataset for use in our input pipeline.
  3. Define the model using the tf.keras model subclassing API.
  4. Train the model using eager execution.
  5. Demonstrate how to use the trained model.

Example #1: Text Generation

Our first example is for text generation, where we use an RNN to generate text in a similar style to Shakespeare. You can run it on Colaboratory with the link above (or you can also download it as a Jupyter notebook from GitHub). The code is explained in detail in the notebook.

Given a large collection of Shakespeare’s writings, this example learns to generate text that sounds and appears similar stylistically:

Example text generated by the notebook after training for 30 epochs on a collection of Shakespeare’s writing.

While most of the sentences will not make sense (of course, this simple model has not learned the meaning of language), what’s impressive is that most of the words *are* valid, and that the structure of the plays it emits look similar to those from the original text. (This is a character based model, in the short amount of time we’ve trained it — it has successfully learned both of those things from scratch). If you like, you can change the dataset by changing a single line of code.

The best place to learn more about RNNs is Andrej Karpathy’s excellent article, The Unreasonable Effectiveness of Recurrent Neural Networks. If you’d like to learn more about implementing RNNs with Keras or tf.keras, we recommend these notebooks by Francois Chollet.

Example #2: DCGAN

In this example, we generate handwritten digits using DCGAN. A Generative Adversarial Network (GAN) consists of a generator and a discriminator. The job of the generator is to create convincing images so as to fool the discriminator. The job of the discriminator is to classify between real images and fake images (created by the generator). The output you see below is generated after training the generator and discriminator for 150 epochs using the architecture and hyperparameters described in this paper.

GIF of images generated every 10 epochs out of 150 epochs. You can find code to create a GIF like this in the notebook.

Example #3: Neural Machine Translation with Attention

This example trains a model to translate Spanish sentences to English sentences. After training the model, you will be able to input a Spanish sentence, such as “¿todavia estan en casa?”, and return the English translation: “are you still at home?”

The image you see below is the attention plot. It shows which parts of the input sentence has the model’s attention while translating. For example, when the model translated the word “cold”, it was looking at “mucho”, “frio”, “aqui”. We implemented Bahdanau Attention from scratch using tf.keras and eager execution, explained in detail in the notebook. You can also use this implementation as a base for implementing you own custom models.

Attention plot for the above translation.

Example #4: Image Captioning with Attention

In this example, we train our model to predict a caption for an image. We also generate an attention plot, which shows the parts of the image the model focuses on as it generates the caption. For example, the model focuses near the surfboard in the image when it predicts the word “surfboard”. This model is trained using a subset of the MS-COCO dataset, which will be downloaded automatically by the notebook.

Predicted Caption for the image.
Attention plot of each word for the above image.

Next steps

To learn more about tf.keras and eager, keep your eyes on tensorflow.org/tutorials for updated content, and periodically check this blog, and TensorFlow’s twitter feed. Thanks for reading!

Acknowledgements

Thanks very much to Josh Gordon, Mark Daoust, Alexandre Passos, Asim Shankar, Billy Lamberta, Daniel ‘Wolff’ Dobson, and Francois Chollet for their contributions and help!

7 Comments

  • yeezy shoes says:

    My spouse and i have been absolutely comfortable when Emmanuel could round up his investigation via the ideas he made in your web site. It’s not at all simplistic just to choose to be freely giving key points that the others might have been selling. And we also fully grasp we now have the blog owner to give thanks to for this. Most of the explanations you have made, the easy website navigation, the relationships you will make it possible to instill – it is everything spectacular, and it’s really aiding our son in addition to our family know that that matter is exciting, which is certainly extremely serious. Thanks for the whole lot!

  • I would like to voice my affection for your kind-heartedness for those who actually need guidance on this concept. Your special dedication to passing the message all through became especially practical and has really empowered girls much like me to reach their desired goals. Your amazing invaluable facts entails a whole lot to me and somewhat more to my colleagues. With thanks; from all of us.

  • ultra boost says:

    My wife and i ended up being so peaceful that Albert could round up his reports out of the ideas he grabbed from your web pages. It’s not at all simplistic to just continually be freely giving guides which often some other people may have been making money from. We recognize we have got the website owner to be grateful to because of that. The type of explanations you’ve made, the easy website menu, the friendships you can aid to engender – it’s everything wonderful, and it is facilitating our son and the family believe that that idea is exciting, which is unbelievably fundamental. Thank you for the whole thing!

  • yeezys says:

    My spouse and i got now more than happy Ervin managed to deal with his reports from your ideas he got through the blog. It is now and again perplexing to just always be offering helpful hints which a number of people have been selling. Therefore we take into account we have the writer to thank for that. The type of explanations you’ve made, the simple website navigation, the relationships you help to instill – it’s most amazing, and it’s really assisting our son and us do think that concept is thrilling, and that’s wonderfully mandatory. Thank you for the whole lot!

  • I want to show thanks to you just for bailing me out of this situation. As a result of searching through the the web and meeting opinions that were not pleasant, I assumed my life was gone. Being alive without the approaches to the issues you’ve sorted out through your good article is a serious case, and ones which may have in a negative way damaged my career if I had not noticed your web blog. Your know-how and kindness in dealing with almost everything was excellent. I don’t know what I would have done if I hadn’t come upon such a subject like this. I can also at this point relish my future. Thanks so much for your impressive and sensible help. I will not be reluctant to propose your web sites to anybody who should receive direction on this subject.

  • nmd says:

    I have to express thanks to the writer for bailing me out of this type of condition. Just after looking through the world wide web and getting things which were not powerful, I assumed my entire life was well over. Existing without the approaches to the difficulties you’ve solved as a result of your entire report is a serious case, as well as the ones which may have negatively damaged my entire career if I had not discovered the blog. Your main expertise and kindness in taking care of all the details was valuable. I’m not sure what I would’ve done if I hadn’t come across such a solution like this. I’m able to now relish my future. Thank you so much for this impressive and sensible help. I will not hesitate to endorse the sites to anybody who needs and wants assistance on this topic.

  • jordan 13 says:

    I really wanted to make a quick word so as to thank you for those awesome concepts you are giving out here. My long internet look up has at the end been compensated with reliable details to talk about with my great friends. I ‘d say that we website visitors actually are undeniably fortunate to dwell in a fantastic community with so many marvellous people with useful basics. I feel very much happy to have discovered your site and look forward to some more fun moments reading here. Thanks once again for all the details.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.