Bidirectional Long Short-Term Memory (BI-LSTM)

YouTube search... ...Google search

• Recurrent Neural Network (RNN) Variants:
  • Long Short-Term Memory (LSTM)
  • Manhattan LSTM (MaLSTM) — a Siamese architecture based on recurrent neural network
  • Gated Recurrent Unit (GRU)
  • Bidirectional Long Short-Term Memory (BI-LSTM)
  • Bidirectional Long Short-Term Memory (BI-LSTM) with Attention Mechanism
  • Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)
  • Hopfield Network (HN)
• Attention Mechanism ...Transformer Model ...Generative Pre-trained Transformer (GPT)

The purpose of the Bi-LSTM is to look at a particular sequences both from front-to-back as well as from back-to-front. In this way, the network creates a context for each character in the text that depends on both its past as well as its future.

Bidirectional Long/Short-Term Memory (BiLSTM) look exactly the same as its unidirectional counterpart. The difference is that the network is not just connected to the past, but also to the future. As an example, unidirectional LSTMs might be trained to predict the word “fish” by being fed the letters one by one, where the recurrent connections through time remember the last value. A BiLSTM would also be fed the next letter in the sequence on the backward pass, giving it access to future information. This trains the network to fill in gaps instead of advancing information, so instead of expanding an image on the edge, it could fill a hole in the middle of an image. Schuster, Mike, and Kuldip K. Paliwal. “Bidirectional recurrent neural networks.” IEEE Transactions on Signal Processing 45.11 (1997): 2673-2681.