Attention

YouTube search... ...Google search

• Memory Networks
• Transformers
• Sequence to Sequence (Seq2Seq)
• Recurrent Neural Networks (RNN)
• Autoencoder (AE) / Encoder-Decoder
• Bidirectional Long Short-Term Memory (BI-LSTM) with Attention Mechanism
• Natural Language Processing (NLP)
• Feature Exploration/Learning
• Attention? Attention! | Lilian Weng
• The Illustrated Transformer | Jay Alammar

Attention mechanisms in neural networks are about memory access. That’s the first thing to remember about attention: it’s something of a misnomer.

Attention networks are a kind of short-term memory that allocates attention over input features they have recently seen. Attention mechanisms are components of memory networks, which focus their attention on external memory storage rather than a sequence of hidden states in a Recurrent Neural Networks (RNN). Memory networks are a little different, but not too. They work with external data storage, and they are useful for, say, mapping questions as input to answers stored in that external memory. That external data storage acts as an embedding that the attention mechanism can alter, writing to the memory what it learns, and reading from it to make a prediction. While the hidden states of a recurrent neural network are a sequence of embeddings, memory is an accumulation of those embeddings (imagine performing max pooling on all your hidden states – that would be like memory). A Beginner's Guide to Attention Mechanisms and Memory Networks | Skymind

3 ways of Attention:

1. Autoencoder (AE) / Encoder-Decoder
2. Encoder Self-Attention
3. MaskedDecoder Self-Attention

Attention Is All You Need

The dominant sequence transduction models are based on complex Recurrent Neural Network (RNN)) or (Deep) Convolutional Neural Network (DCNN/CNN) in an encoder-decoder (Autoencoder (AE) / Encoder-Decoder} configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Attention Is All You Need | A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polosukhin - Google

Making decisions about where to send information

Making decisions about where to send information. An AI Pioneer Explains The Evolution Of Neural Networks | Nichokas Thompson - Wired

• Capsule Networks (CapNets)