Gradient Descent Optimization & Challenges
YouTube search... ...Google search
 Gradient Boosting Algorithms
 Backpropagation
 Objective vs. Cost vs. Loss vs. Error Function
 Topology and Weight Evolving Artificial Neural Network (TWEANN)
 What is Gradient Descent?  Daniel Nelson  Unite.ai
 Other Challenges in Artificial Intelligence
 AverageStochastic Gradient Descent (SGD) WeightDropped LSTM (AWDLSTM)
Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. In machine learning, we use gradient descent to update the parameters of our model. Parameters refer to coefficients in Linear Regression and weights in neural networks. Gradient Descent  ML Cheatsheet
Gradient Descent is the most common optimization algorithm in machine learning and deep learning. It is a firstorder optimization algorithm. This means it only takes into account the first derivative when performing the updates on the parameters. On each iteration, we update the parameters in the opposite direction of the gradient of the objective function J(w) w.r.t the parameters where the gradient gives the direction of the steepest ascent. Gradient Descent Algorithm and Its Variants  Imad Dabbura  Towards Data Science
Nonlinear Regression algorithms, which fit curves that are not linear in their parameters to data, are a little more complicated, because, unlike linear Regression problems, they can’t be solved with a deterministic method. Instead, the nonlinear Regression algorithms implement some kind of iterative minimization process, often some variation on the method of steepest descent. Steepest descent basically computes the squared error and its gradient at the current parameter values, picks a step size (aka learning rate), follows the direction of the gradient “down the hill,” and then recomputes the squared error and its gradient at the new parameter values. Eventually, with luck, the process converges. The variants on steepest descent try to improve the convergence properties. Machine learning algorithms are even less straightforward than nonlinear Regression, partly because machine learning dispenses with the constraint of fitting to a specific mathematical function, such as a polynomial. There are two major categories of problems that are often solved by machine learning: Regression and classification. Regression is for numeric data (e.g. What is the likely income for someone with a given address and profession?) and classification is for nonnumeric data (e.g. Will the applicant default on this loan?). Machine learning algorithms explained  Martin Heller  InfoWorld
Contents
Gradient Descent  Stochastic (SGD), Batch (BGD) & MiniBatch
Youtube search... ...Google search
Learning Rate Decay
YouTube search... ...Google search
Adapting the learning rate for your stochastic gradient descent optimization procedure can increase performance and reduce training time. Sometimes this is called learning rate annealing or adaptive learning rates. The simplest and perhaps most used adaptation of learning rate during training are techniques that reduce the learning rate over time. These have the benefit of making large changes at the beginning of the training procedure when larger learning rate values are used, and decreasing the learning rate such that a smaller rate and therefore smaller training updates are made to weights later in the training procedure. This has the effect of quickly learning good weights early and fine tuning them later. The 10 Deep Learning Methods AI Practitioners Need to Apply  James Le
Two popular and easy to use learning rate decay are as follows:
 Decrease the learning rate gradually based on the epoch.
 Decrease the learning rate using punctuated large drops at specific epochs.


Gradient Descent With Momentum
YouTube search... ...Google search
helps accelerate gradients vectors in the right directions, thus leading to faster converging. It is one of the most popular optimization algorithms and many stateoftheart models are trained using it... With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. Which means we’re not always going in the optimal direction, because our derivatives are ‘noisy’. Just like in my graphs above. So, exponentially weighed averages can provide us a better estimate which is closer to the actual derivate than our noisy calculations. This is one reason why momentum might work better than classic SGD. The other reason lies in ravines. Ravine is an area, where the surface curves much more steeply in one dimension than in another. Ravines are common near local minimas in deep learning and SGD has troubles navigating them. SGD will tend to oscillate across the narrow ravine since the negative gradient will point down one of the steep sides rather than along the ravine towards the optimum. Momentum helps accelerate gradients in the right direction. This is expressed in the following pictures: Stochastic Gradient Descent with momentum  Vitaly Bushaev  Towards Data Science


Vanishing & Exploding Gradients Problems
YouTube Search ...Google search
Vanishing & Exploding Gradients Challenges with Long ShortTerm Memory (LSTM) and Recurrent Neural Networks (RNN)
YouTube Search ...Google search