Gradient Descent Optimization & Challenges

YouTube search... ...Google search

• Gradient Boosting Algorithms
• Backpropagation ...Gradient Descent Optimization & Challenges ...Feed Forward Neural Network (FF or FFNN) ...Forward-Forward
• Objective vs. Cost vs. Loss vs. Error Function
• Topology and Weight Evolving Artificial Neural Network (TWEANN)
• What is Gradient Descent? | Daniel Nelson - Unite.ai
• Other Challenges in Artificial Intelligence
• Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)

Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. In machine learning, we use gradient descent to update the parameters of our model. Parameters refer to coefficients in Linear Regression and weights in neural networks. Gradient Descent | ML Cheatsheet

Gradient Descent is the most common optimization algorithm in machine learning and deep learning. It is a first-order optimization algorithm. This means it only takes into account the first derivative when performing the updates on the parameters. On each iteration, we update the parameters in the opposite direction of the gradient of the objective function J(w) w.r.t the parameters where the gradient gives the direction of the steepest ascent. Gradient Descent Algorithm and Its Variants | Imad Dabbura - Towards Data Science

Nonlinear Regression algorithms, which fit curves that are not linear in their parameters to data, are a little more complicated, because, unlike linear Regression problems, they can’t be solved with a deterministic method. Instead, the nonlinear Regression algorithms implement some kind of iterative minimization process, often some variation on the method of steepest descent. Steepest descent basically computes the squared error and its gradient at the current parameter values, picks a step size (aka learning rate), follows the direction of the gradient “down the hill,” and then recomputes the squared error and its gradient at the new parameter values. Eventually, with luck, the process converges. The variants on steepest descent try to improve the convergence properties. Machine learning algorithms are even less straightforward than nonlinear Regression, partly because machine learning dispenses with the constraint of fitting to a specific mathematical function, such as a polynomial. There are two major categories of problems that are often solved by machine learning: Regression and classification. Regression is for numeric data (e.g. What is the likely income for someone with a given address and profession?) and classification is for non-numeric data (e.g. Will the applicant default on this loan?). Machine learning algorithms explained | Martin Heller - InfoWorld

Gradient Descent - Stochastic (SGD), Batch (BGD) & Mini-Batch

Youtube search... ...Google search

Learning Rate Decay

YouTube search... ...Google search

Adapting the learning rate for your stochastic gradient descent optimization procedure can increase performance and reduce training time. Sometimes this is called learning rate annealing or adaptive learning rates. The simplest and perhaps most used adaptation of learning rate during training are techniques that reduce the learning rate over time. These have the benefit of making large changes at the beginning of the training procedure when larger learning rate values are used, and decreasing the learning rate such that a smaller rate and therefore smaller training updates are made to weights later in the training procedure. This has the effect of quickly learning good weights early and fine tuning them later. The 10 Deep Learning Methods AI Practitioners Need to Apply | James Le

Two popular and easy to use learning rate decay are as follows:

• Decrease the learning rate gradually based on the epoch.
• Decrease the learning rate using punctuated large drops at specific epochs.

Learning Rate Decay (C2W2L09) Deeplearning.ai Andrew Ng Take the Deep Learning Specialization: https://bit.ly/2Tx5XGn Check out all our courses: https://www.deeplearning.ai Subscribe to The Batch, our weekly newsletter: https://www.deeplearning.ai/thebatch

Learning Rate in a Neural Network explained In this video, we explain the concept of the learning rate used during training of an artificial neural network and also show how to specify the learning rate in code with Keras. Hey, we're Chris and Mandy, the creators of deeplizard! https://youtube.com/deeplizardvlog

Gradient Descent With Momentum

YouTube search... ...Google search

helps accelerate gradients vectors in the right directions, thus leading to faster converging. It is one of the most popular optimization algorithms and many state-of-the-art models are trained using it... With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. Which means we’re not always going in the optimal direction, because our derivatives are ‘noisy’. Just like in my graphs above. So, exponentially weighed averages can provide us a better estimate which is closer to the actual derivate than our noisy calculations. This is one reason why momentum might work better than classic SGD. The other reason lies in ravines. Ravine is an area, where the surface curves much more steeply in one dimension than in another. Ravines are common near local minimas in deep learning and SGD has troubles navigating them. SGD will tend to oscillate across the narrow ravine since the negative gradient will point down one of the steep sides rather than along the ravine towards the optimum. Momentum helps accelerate gradients in the right direction. This is expressed in the following pictures: Stochastic Gradient Descent with momentum | Vitaly Bushaev - Towards Data Science

Gradient Descent With Momentum (C2W2L06) Deeplearning.ai Andrew Ng Take the Deep Learning Specialization: https://bit.ly/2Tx5XGn Check out all our courses: https://www.deeplearning.ai Subscribe to The Batch, our weekly newsletter: https://www.deeplearning.ai/thebatch

23. Accelerating Gradient Descent (Use Momentum) MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Spring 2018 Instructor: Gilbert Strang In this lecture, Professor Strang explains both momentum-based gradient descent and Nesterov's accelerated gradient descent. View the complete course: https://ocw.mit.edu/18-065S18

Vanishing & Exploding Gradients Problems

YouTube Search ...Google search

• Vanishing gradient problem

Vanishing & Exploding Gradients Challenges with Long Short-Term Memory (LSTM) and Recurrent Neural Networks (RNN)

YouTube Search ...Google search

• Recurrent Neural Network (RNN)