Optimization Methods

Youtube search... ...Google search

• Natural Language Processing (NLP)
• Recurrent Neural Network (RNN)
  • Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)
• Gradient Boosting Algorithms

Methods:

• Stochastic gradient descent (SGD) (with and without momentum)
• L-BGFS
• Adagrad
• Adadelta
• Root Mean Squared (RMSprop)
• Adam
• Hessian-free (HF)