Hopfield Network (HN)

YouTube search... ...Google search

• Neural Network Zoo | Fjodor Van Veen
• Hopfield Networks | Chris Nicholson - A.I. Wiki pathmind
• Recurrent Neural Network (RNN) Variants:
  • Long Short-Term Memory (LSTM)
  • Gated Recurrent Unit (GRU)
  • Bidirectional Long Short-Term Memory (BI-LSTM)
  • Bidirectional Long Short-Term Memory (BI-LSTM) with Attention Mechanism
  • Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)
  • Hopfield Network (HN)

A Hopfield Network is a form (one particular type) of recurrent artificial neural network popularized by John Hopfield in 1982, but described earlier by Little in 1974. Hopfield networks serve as content-addressable ("associative") memory systems with binary threshold nodes. They are guaranteed to converge to a local minimum and, therefore, may converge to a false pattern (wrong local minimum) rather than the stored pattern (expected local minimum). Wikipedia

Hopfield Network the weight from node to another and from the later to the former are the same (symmetric). The Hopfield Network is fully connected, so every neuron’s output is an input to all the other neurons. Another feature of the network is that updating of nodes happens in a binary way. These features allow for a particular feature of Hopfield's nets - they are guaranteed to converge to an attractor (stable state). oba2311

Every neuron is connected to every other neuron; it is a completely entangled plate of spaghetti as even all the nodes function as everything. Each node is input before training, then hidden during training and output afterwards. The networks are trained by setting the value of the neurons to the desired pattern after which the weights can be computed. The weights do not change after this. Once trained for one or more patterns, the network will always converge to one of the learned patterns because the network is only stable in those states. Note that it does not always conform to the desired state (it’s not a magic black box sadly). It stabilizes in part due to the total “energy” or “temperature” of the network being reduced incrementally during training. Each neuron has an activation threshold which scales to this temperature, which if surpassed by summing the input causes the neuron to take the form of one of two states (usually -1 or 1, sometimes 0 or 1). Updating the network can be done synchronously or more commonly one by one. If updated one by one, a fair random sequence is created to organize which cells update in what order (fair random being all options (n) occurring exactly once every n items). This is so you can tell when the network is stable (done converging), once every cell has been updated and none of them changed, the network is stable (annealed). These networks are often called associative memory because the converge to the most similar state as the input; if humans see half a table we can image the other half, this network will converge to a table if presented with half noise and half a table. Hopfield, John J. “Neural networks and physical systems with emergent collective computational abilities.” Proceedings of the national academy of sciences 79.8 (1982): 2554-2558.