Difference between revisions of "Activation Functions"
| Line 10: | Line 10: | ||
<youtube>I2rMl0Ea5UE</youtube> | <youtube>I2rMl0Ea5UE</youtube> | ||
| + | * [https://www.tensorflow.org/api_guides/python/nn TensorFlow Python API] | ||
| + | The activation ops provide different types of nonlinearities for use in neural networks. These include smooth nonlinearities (sigmoid, tanh, elu, selu, softplus, and softsign), continuous but not everywhere differentiable functions (relu, relu6, crelu and relu_x), and random regularization (dropout). All activation ops apply componentwise, and produce a tensor of the same shape as the input tensor. | ||
| − | === tanh === | + | * [http://stackoverflow.com/questions/39921607/how-to-make-a-custom-activation-function-with-only-python-in-tensorflow Roll your own TensorFlow activation function] |
| + | |||
| + | * [https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6 Explained] | ||
| + | |||
| + | === tanh (hyperbolic tangent) === | ||
<youtube>ZyThOXVZXgc</youtube> | <youtube>ZyThOXVZXgc</youtube> | ||
| − | === ReLU === | + | === ReLU (Rectified Linear Unit) === |
<youtube>pWxSUZWOctk</youtube> | <youtube>pWxSUZWOctk</youtube> | ||
<youtube>Opg63pan_YQ</youtube> | <youtube>Opg63pan_YQ</youtube> | ||
| + | |||
| + | === Leaky ReLU (Rectified Linear Unit) === | ||
=== Softmax === | === Softmax === | ||
| Line 30: | Line 38: | ||
<youtube>0RTvFKCQ_yo</youtube> | <youtube>0RTvFKCQ_yo</youtube> | ||
| − | === | + | |
| + | === Threshold (binary step) ...not offered in TensorFlow library === | ||
| + | |||
| + | <youtube>https://www.youtube.com/watch?v=Kd7gDHY_OUU</youtube> | ||
| + | |||
| + | === Piecewise === | ||
| + | |||
| + | <youtube>aHQgFpJ_xp8</youtube> | ||
| + | |||
| + | === Identity (linear) === | ||
<youtube>aHQgFpJ_xp8</youtube> | <youtube>aHQgFpJ_xp8</youtube> | ||
Revision as of 10:46, 5 May 2018
The activation ops provide different types of nonlinearities for use in neural networks. These include smooth nonlinearities (sigmoid, tanh, elu, selu, softplus, and softsign), continuous but not everywhere differentiable functions (relu, relu6, crelu and relu_x), and random regularization (dropout). All activation ops apply componentwise, and produce a tensor of the same shape as the input tensor.
Contents
tanh (hyperbolic tangent)
ReLU (Rectified Linear Unit)
Leaky ReLU (Rectified Linear Unit)
Softmax
Sigmoid
Threshold (binary step) ...not offered in TensorFlow library
Piecewise
Identity (linear)
