Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)

Youtube search... ...Google search

• Natural Language Processing (NLP)
• Recurrent Neural Network (RNN) Variants:
  • Long Short-Term Memory (LSTM)
  • Gated Recurrent Unit (GRU)
  • Bidirectional Long Short-Term Memory (BI-LSTM)
  • Bidirectional Long Short-Term Memory (BI-LSTM) with Attention Mechanism
  • Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)
  • Hopfield Network (HN)
• Optimization Methods

The AWD-LSTM has been dominating the state-of-the-art language modeling. All the top research papers on word-level models incorporate AWD-LSTMs. And it has shown great results on character-level models as well. ..The AWD-LSTM stands for ASGD Weight-Dropped LSTM. It uses DropConnect and a variant of Average-SGD (NT-ASGD) along with several other well-known regularization strategies. What makes the AWD-LSTM great? | Yashu Seth