Difference between revisions of "Autoencoder (AE) / Encoder-Decoder"
m (BPeat moved page Autoencoder / Encoder-Decoder to Autoencoder (AE) / Encoder-Decoder without leaving a redirect) |
|||
| Line 5: | Line 5: | ||
*[[Attention Models]] | *[[Attention Models]] | ||
*[[Natural Language Inference (NLI) and Recognizing Textual Entailment (RTE)]] | *[[Natural Language Inference (NLI) and Recognizing Textual Entailment (RTE)]] | ||
| + | |||
| + | Autoencoders (AE) are somewhat similar to Feed forward neural networks (FFNNs) as AEs are more like a different use of FFNNs than a fundamentally different architecture. The basic idea behind autoencoders is to encode information (as in compress, not encrypt) automatically, hence the name. The entire network always resembles an hourglass like shape, with smaller hidden layers than the input and output layers. AEs are also always symmetrical around the middle layer(s) (one or two depending on an even or odd amount of layers). The smallest layer(s) is|are almost always in the middle, the place where the information is most compressed (the chokepoint of the network). Everything up to the middle is called the encoding part, everything after the middle the decoding and the middle (surprise) the code. One can train them using backpropagation by feeding input and setting the error to be the difference between the input and what came out. AEs can be built symmetrically when it comes to weights as well, so the encoding weights are the same as the decoding weights. Bourlard, Hervé, and Yves Kamp. “Auto-association by multilayer perceptrons and singular value decomposition.” Biological cybernetics 59.4-5 (1988): 291-294. | ||
<youtube>H1AllrJ-_30</youtube> | <youtube>H1AllrJ-_30</youtube> | ||
Revision as of 20:45, 11 May 2018
- Neural Network Zoo | Fjodor Van Veen
- Sequence to Sequence (Seq2Seq)
- Attention Models
- Natural Language Inference (NLI) and Recognizing Textual Entailment (RTE)
Autoencoders (AE) are somewhat similar to Feed forward neural networks (FFNNs) as AEs are more like a different use of FFNNs than a fundamentally different architecture. The basic idea behind autoencoders is to encode information (as in compress, not encrypt) automatically, hence the name. The entire network always resembles an hourglass like shape, with smaller hidden layers than the input and output layers. AEs are also always symmetrical around the middle layer(s) (one or two depending on an even or odd amount of layers). The smallest layer(s) is|are almost always in the middle, the place where the information is most compressed (the chokepoint of the network). Everything up to the middle is called the encoding part, everything after the middle the decoding and the middle (surprise) the code. One can train them using backpropagation by feeding input and setting the error to be the difference between the input and what came out. AEs can be built symmetrically when it comes to weights as well, so the encoding weights are the same as the decoding weights. Bourlard, Hervé, and Yves Kamp. “Auto-association by multilayer perceptrons and singular value decomposition.” Biological cybernetics 59.4-5 (1988): 291-294.
