Only Numpy: Deriving Forward Feed on Multi-Dimensional Recurrent Neural Networks (Spatial LSTM) by “Generative Image Modeling Using Spatial LSTMs”

By Jae Duk Seo

Multi-Dimensional Recurrent Neural Networks, I became interested in them as soon as I heard it’s name. So today, I will attempt to tackle the network structure of Spatial LSTM introduce in this paper. “ Generative Image Modeling Using Spatial LSTMs” — by Lucas Theis. Also for today’s blog we will perform Forward Feed on 2D LSTM.

Transform from 1D LSTM to 2D LSTM

So above image shows how we can take the idea of 1D LSTM, to 2D LSTM. To apply them on images. One very important thing to take note from above photo are the Cell State and hidden States.

Yellow Box → 1D LSTM
Green Box → Transposed 1D LSTM 
(Think about it as being one column in a matrix)
Pink Box → 2D LSTM

1D LSTM that depends on Time

As seen above, for 1D LSTM, we initialize C(0) and h(0) before we start to train the network. There are multiple of methods to initialize these values, for example in the paper “ Show, Attend and Tell: Neural Image Caption Generation with Visual Attention” the authors initialize the first values via something called MLP — I can only assume that it is Multi Layer Perceptrons.

But in 2D LSTM, we have to initialize whole lot more of cell and hidden state values.

2D LSTM respect to time

As seen above, not only we need to initialize from C(0,1) to C(0,j) but also C(1,0) to C(i,0). Same goes for all hidden states. Now we can do something interesting, since we know the structure of 1D LSTM and 2D LSTM, let’s imagine 3D LSTM.


Quite a beauty isn’t she? 😀 
Again, the orange boxes are the location of the first Cell and Hidden States. The applications for this network is not only bounding to video data but much more. Now we know the general structure, lets go back to the paper “ Generative Image Modeling Using Spatial LSTMs

Spatial long short-term memory

Image from original paper

So as the authors said, the original SLSTM was proposed by the two authors Graves & Schmidhuber. To see the paper by those two authors please click “ Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks”. In that paper, the authors have a very good visual of what an 2D LSTM is and it is shown below. However, the paper that I am working with have more clear and clean mathematical equation that describes SLSTM. (Shown above)

Sample Training Data

So we will do forward feed pass on a VERY simple training data, which is an image that have dimension of 2*2 (total of 4 pixel), shown in the black box above.

Network Architecture

Now I know that it looks bad, but I had to use the whole white board to make that diagram LOL so work with me here. Lets start from the beginning.

First each box represents one LSTM box, the architecture is a derivation from the famous Colah Blog.

Image from Colah Blog

Second, here is the time stamp information below.

Red Box → Forward Feed when time stamp is (1,1)
Green Box → Forward Feed when time stamp is (2,1)
Orange Box → Forward Feed when time stamp is (1,2)
Purple Box → Forward Feed when time stamp is (2,2)

Third, each blue star represents the cost function we can calculate at each time stamp.

Forward Feed

Again, I know that it looks bad, but with LSTM’s the equations get messy all of the time.

One thing to note is all of the variables written with BLUE markers are already initialized values. So don’t worry about where they just popped up from no where, they were initialized before hand.

Detailed Look at Forward Feed at Time Stamp (1,1) and (1,2)
Detailed Look at Forward Feed at Time Stamp (2,1) and (2,2)

Final Words

I can’t image the back propagation process for this network, it will be SO fun to derive them by hand. I’ll hopefully do that one day.

If any errors are found, please email me at

Meanwhile follow me on my twitter here, and visit my website, or my Youtube channel for more content. I also did deriving back propagation on simple RNN here if you are interested.


  1. Theis, L., & Bethge, M. (2015). Generative image modeling using spatial LSTMs. In Advances in Neural Information Processing Systems (pp. 1927–1935).
  2. CoRR, abs/1502.03044, . Kelvin Xu and (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual.
  3. CoRR, abs/0705.2011, . Alex Graves and (2007). Multi-Dimensional Recurrent Neural Networks.
  4. Understanding LSTM Networks. (n.d.). Retrieved January 19, 2018, from
  5. Graves, A., & Schmidhuber, J. (2009). Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in neural information processing systems (pp. 545–552).
0 0 votes
Article Rating
Notify of
Newest Most Voted
Inline Feedbacks
View all comments
bape hoodie
3 years ago

I wanted to send a quick remark so as to express gratitude to you for all of the superb ideas you are placing at this website. My particularly long internet investigation has at the end been compensated with good quality insight to write about with my co-workers. I would assert that most of us visitors actually are extremely endowed to live in a magnificent website with very many wonderful people with beneficial hints. I feel very much happy to have encountered the web site and look forward to really more awesome minutes reading here. Thank you again for everything.

3 years ago

You’ve really helped me unearstdnd the issues. Thanks.

Adidas NMD Runner PK Black Blue/Red

My husband and i felt peaceful that Edward could conclude his inquiry because of the ideas he received when using the site. It is now and again perplexing to just happen to be making a gift of strategies that many people have been making money from. So we keep in mind we need the blog owner to appreciate for this. The explanations you’ve made, the straightforward blog menu, the relationships your site aid to promote – it’s got all fantastic, and it is leading our son and the family believe that the topic is amusing, and that is rather mandatory.… Read more »

Adidas NMD R1 Primeknit Core Black Boost Japan Pack

My spouse and i ended up being absolutely relieved when Raymond managed to do his web research with the ideas he gained from your own web page. It is now and again perplexing just to choose to be offering procedures which often the others could have been making money from. And we do understand we have you to give thanks to because of that. The main explanations you’ve made, the easy blog navigation, the relationships your site make it possible to promote – it’s got many impressive, and it’s letting our son in addition to us reason why the idea… Read more »

adidas ultra boost uncaged

I have to show some thanks to you just for rescuing me from this difficulty. Because of researching throughout the internet and obtaining ways that were not helpful, I assumed my entire life was done. Living without the solutions to the problems you’ve resolved by means of your posting is a critical case, and ones that could have negatively affected my entire career if I hadn’t discovered your web page. Your main ability and kindness in dealing with all areas was invaluable. I am not sure what I would have done if I had not come upon such a stuff… Read more »

Adidas NMD x Supreme Black

I want to express my thanks to you for rescuing me from this particular issue. After searching throughout the internet and obtaining principles that were not pleasant, I figured my life was over. Existing without the presence of solutions to the issues you’ve solved by way of your main short post is a crucial case, as well as the kind which may have badly damaged my entire career if I had not noticed your blog. Your own personal mastery and kindness in controlling everything was crucial. I don’t know what I would have done if I had not come upon… Read more »

yeezy boost 350
3 years ago

I truly wanted to send a brief note to be able to appreciate you for some of the fabulous tips you are posting at this site. My considerable internet search has at the end been paid with sensible concept to write about with my friends and classmates. I would suppose that most of us visitors are undoubtedly lucky to dwell in a very good site with so many wonderful individuals with valuable secrets. I feel very grateful to have used your web site and look forward to tons of more fun times reading here. Thanks a lot once again for… Read more »

Adidas Originals NMD X Yeezy 350 Boost Mens Running Shoes Dark Gray

I simply had to thank you very much again. I am not sure what I would have made to happen without the entire creative ideas revealed by you regarding this area of interest. It was before the difficult circumstance in my position, nevertheless viewing a specialised way you dealt with it forced me to weep for fulfillment. I am happy for the guidance and then pray you comprehend what a great job you were accomplishing teaching many others all through your blog. I’m certain you have never met all of us.

Adidas NMD PK Runner Deep Blue men women shoe

I have to express appreciation to this writer just for rescuing me from such a situation. After surfing around throughout the world-wide-web and seeing ideas which were not powerful, I assumed my life was gone. Being alive without the presence of approaches to the difficulties you have resolved as a result of this site is a serious case, and those that would have negatively damaged my entire career if I hadn’t noticed your web page. That competence and kindness in taking care of all the stuff was vital. I am not sure what I would have done if I had… Read more »

off white clothing
3 years ago

My spouse and i ended up being very joyful when Louis could complete his inquiry through your precious recommendations he obtained from your own site. It’s not at all simplistic to simply always be releasing facts which some other people may have been making money from. We do understand we’ve got the blog owner to give thanks to because of that. The main explanations you’ve made, the easy site menu, the relationships you can assist to instill – it’s got everything fantastic, and it’s really aiding our son and our family reckon that this issue is entertaining, and that’s seriously… Read more »

Neue Bayern München Fußballtrikot

Amazing blog! Do you have any tips for aspiring writers?
I’m planning to start my own website soon but I’m a little
lost on everything. Would you recommend starting with a free platform like WordPress or go for a paid option? There are so many options out there that I’m totally confused ..

Any recommendations? Thanks a lot! Neue Bayern München Fußballtrikot

2 years ago

Hi every one, here every person is sharing these kinds of knowledge, so it’s nice to read this webpage, and I
used to go to see this weblog daily.

2 years ago

What’s up, after reading this amazing paragraph i am also delighted to share my knowledge here with friends.