XLNet - Revision history

BPeat at 01:08, 10 July 2023

2023-07-10T01:08:28Z

BPeat at 01:36, 18 May 2023

2023-05-18T01:36:39Z

BPeat at 19:56, 28 April 2023

2023-04-28T19:56:52Z

BPeat at 15:25, 19 March 2023

2023-03-19T15:25:10Z

BPeat at 12:32, 9 June 2020

2020-06-09T12:32:03Z

BPeat at 12:31, 9 June 2020

2020-06-09T12:31:22Z

BPeat at 12:30, 9 June 2020

2020-06-09T12:30:31Z

BPeat at 10:04, 10 July 2019

2019-07-10T10:04:17Z

BPeat at 10:03, 10 July 2019

2019-07-10T10:03:40Z

BPeat at 09:59, 10 July 2019

2019-07-10T09:59:22Z

@@ Line 12: / Line 12: @@
 * [http://towardsdatascience.com/what-is-xlnet-and-why-it-outperforms-bert-8d8fce710335 What is XLNet and why It outperforms BERT | BrambleXu - Towards Data Science]
 * [http://mlexplained.com/2019/06/30/paper-dissected-xlnet-generalized-autoregressive-pretraining-for-language-understanding-explained/ Paper Dissected: “XLNet: Generalized Autoregressive Pretraining for Language Understanding” Explained | Machine Learning Explained]
-* [[Natural Language Processing (NLP)]]
+* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ... [[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ...  [[Natural Language Tools & Services|Tools & Services]]
 * XLNet combines the best of:
 ** [[Autoregressive]]; such as [[Bidirectional Long Short-Term Memory (BI-LSTM)]], but XLNet uses Self-[[Attention]] instead of [[Long Short-Term Memory (LSTM)]] and 'permutations' on both sides of the bidirectional scan
@@ Line 20: / Line 20: @@
-XLNet is a new unsupervised language representation learning method based on a novel generalized permutation language modeling objective. Additionally, XLNet employs [[Transformer-XL]] as the backbone model, exhibiting excellent performance for language tasks involving long [[context]]. Overall, XLNet achieves state-of-the-art (SOTA) results on various downstream language tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://github.com/zihangdai/xlnet XLNet | zihangdai - GitHub]
+XLNet is a new unsupervised language representation learning method based on a novel generalized permutation language modeling objective. Additionally, XLNet employs [[Transformer-XL]] as the backbone model, exhibiting excellent performance for language tasks involving long [[context]]. Overall, XLNet achieves state-of-the-art (SOTA) results on various downstream language tasks including question answering, natural language inference, [[Sentiment Analysis]], and document ranking. [http://github.com/zihangdai/xlnet XLNet | zihangdai - GitHub]
-With the capability of modeling bidirectional [[context]]s, denoising autoencoding based pretraining like [[Bidirectional Encoder Representations from Transformers (BERT)]] achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, [[Bidirectional Encoder Representations from Transformers (BERT)]] neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional [[context]]s by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of [[Bidirectional Encoder Representations from Transformers (BERT)]] thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from [[Transformer-XL]], the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://arxiv.org/abs/1906.08237 : Generalized Autoregressive Pretraining for Language Understanding | Z. Yang, Z. Dai, Y Yang, J. Carbonell, R. Salakhutdinov, and Q Le]
+With the capability of modeling bidirectional [[context]]s, denoising autoencoding based pretraining like [[Bidirectional Encoder Representations from Transformers (BERT)]] achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, [[Bidirectional Encoder Representations from Transformers (BERT)]] neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional [[context]]s by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of [[Bidirectional Encoder Representations from Transformers (BERT)]] thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from [[Transformer-XL]], the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, [[Sentiment Analysis]], and document ranking. [http://arxiv.org/abs/1906.08237 : Generalized Autoregressive Pretraining for Language Understanding | Z. Yang, Z. Dai, Y Yang, J. Carbonell, R. Salakhutdinov, and Q Le]
 http://cdn-images-1.medium.com/max/1600/1*bmSZYhV6XlzRFcStRs1iDw.png

@@ Line 20: / Line 20: @@
-XLNet is a new unsupervised language representation learning method based on a novel generalized permutation language modeling objective. Additionally, XLNet employs [[Transformer-XL]] as the backbone model, exhibiting excellent performance for language tasks involving long context. Overall, XLNet achieves state-of-the-art (SOTA) results on various downstream language tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://github.com/zihangdai/xlnet XLNet | zihangdai - GitHub]
+XLNet is a new unsupervised language representation learning method based on a novel generalized permutation language modeling objective. Additionally, XLNet employs [[Transformer-XL]] as the backbone model, exhibiting excellent performance for language tasks involving long [[context]]. Overall, XLNet achieves state-of-the-art (SOTA) results on various downstream language tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://github.com/zihangdai/xlnet XLNet | zihangdai - GitHub]
-With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like [[Bidirectional Encoder Representations from Transformers (BERT)]] achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, [[Bidirectional Encoder Representations from Transformers (BERT)]] neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of [[Bidirectional Encoder Representations from Transformers (BERT)]] thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from [[Transformer-XL]], the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://arxiv.org/abs/1906.08237 : Generalized Autoregressive Pretraining for Language Understanding | Z. Yang, Z. Dai, Y Yang, J. Carbonell, R. Salakhutdinov, and Q Le]
+With the capability of modeling bidirectional [[context]]s, denoising autoencoding based pretraining like [[Bidirectional Encoder Representations from Transformers (BERT)]] achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, [[Bidirectional Encoder Representations from Transformers (BERT)]] neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional [[context]]s by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of [[Bidirectional Encoder Representations from Transformers (BERT)]] thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from [[Transformer-XL]], the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking. [http://arxiv.org/abs/1906.08237 : Generalized Autoregressive Pretraining for Language Understanding | Z. Yang, Z. Dai, Y Yang, J. Carbonell, R. Salakhutdinov, and Q Le]
 http://cdn-images-1.medium.com/max/1600/1*bmSZYhV6XlzRFcStRs1iDw.png
@@ Line 35: / Line 35: @@
 http://cdn-images-1.medium.com/max/800/1*QJm6dOSFQF5ahTLZ2WFkHQ.png
-... in order for us to capture the entire bidirectional context in the prediction of the word “times” we need more subsets of the bidirectional context to train the model on, where the word does not see itself..  addresses this by cleverly choosing such subset contexts like the handcrafted one above such that a word does not “see itself” when it is being predicted. ...In summary, by predicting a word at a position in a sentence by using a subset of its bidirectional context, and taking into account the word is itself not seen using the permutation scan rule above, we can avoid a word from seeing itself in a multilayered context. This scheme, however requires that we sample enough permutations so that we make full use of the bidirectional context. [http://towardsdatascience.com/xlnet-a-clever-language-modeling-solution-ab41e87798b0 XLNet — a clever language modeling solution | Ajit Rajasekharan - Medium]
+... in order for us to capture the entire bidirectional [[context]] in the prediction of the word “times” we need more subsets of the bidirectional [[context]] to train the model on, where the word does not see itself..  addresses this by cleverly choosing such subset [[context]]s like the handcrafted one above such that a word does not “see itself” when it is being predicted. ...In summary, by predicting a word at a position in a sentence by using a subset of its bidirectional [[context]], and taking into account the word is itself not seen using the permutation scan rule above, we can avoid a word from seeing itself in a multilayered [[context]]. This scheme, however requires that we sample enough permutations so that we make full use of the bidirectional [[context]]. [http://towardsdatascience.com/xlnet-a-clever-language-modeling-solution-ab41e87798b0 XLNet — a clever language modeling solution | Ajit Rajasekharan - Medium]

@@ Line 8: / Line 8: @@
 [http://www.google.com/search?q=XLNet+natural+language ...Google search]
-* [[Natural Language Processing (NLP)]]
+* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]]  ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ...  [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
 * [http://arxiv.org/pdf/1906.08237.pdf XLNet: Generalized Autoregressive Pretraining for Language Understanding, Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. Le, June 2019]
 * [http://towardsdatascience.com/what-is-xlnet-and-why-it-outperforms-bert-8d8fce710335 What is XLNet and why It outperforms BERT | BrambleXu - Towards Data Science]

@@ Line 15: / Line 15: @@
 * XLNet combines the best of:
 ** [[Autoregressive]]; such as [[Bidirectional Long Short-Term Memory (BI-LSTM)]], but XLNet uses Self-[[Attention]] instead of [[Long Short-Term Memory (LSTM)]] and 'permutations' on both sides of the bidirectional scan
 ** [[Autoencoder (AE) / Encoder-Decoder]] such as autoencoding denoiser of [[Bidirectional Encoder Representations from Transformers (BERT)]] by integrating methods from [[Transformer-XL]]
 * [http://www.topbots.com/leading-nlp-language-models-2020/ 7 Leading Language Models for NLP in 2020 | Mariya Yao - Topbots]

@@ Line 8: / Line 8: @@
 [http://www.google.com/search?q=XLNet+natural+language ...Google search]
 * [http://arxiv.org/pdf/1906.08237.pdf XLNet: Generalized Autoregressive Pretraining for Language Understanding, Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. Le, June 2019]
 * [http://towardsdatascience.com/what-is-xlnet-and-why-it-outperforms-bert-8d8fce710335 What is XLNet and why It outperforms BERT | BrambleXu - Towards Data Science]

← Older revision		Revision as of 12:31, 9 June 2020
Line 23:		Line 23:

	http://cdn-images-1.medium.com/max/1600/1*bmSZYhV6XlzRFcStRs1iDw.png		http://cdn-images-1.medium.com/max/1600/1*bmSZYhV6XlzRFcStRs1iDw.png
−

	<img src="http://cdn-images-1.medium.com/max/1600/1*RGdAU7tXXKqckbDjhoIWmQ.png" width="800" height="600">		<img src="http://cdn-images-1.medium.com/max/1600/1*RGdAU7tXXKqckbDjhoIWmQ.png" width="800" height="600">

@@ Line 26: / Line 26: @@
 <youtube>naOuE9gLbZo</youtube>
 <youtube>Ubh03Sh45b0</youtube>
-<youtube>H5vpBCLo74U</youtube>
 http://cdn-images-1.medium.com/max/800/1*QJm6dOSFQF5ahTLZ2WFkHQ.png
 ... in order for us to capture the entire bidirectional context in the prediction of the word “times” we need more subsets of the bidirectional context to train the model on, where the word does not see itself..  addresses this by cleverly choosing such subset contexts like the handcrafted one above such that a word does not “see itself” when it is being predicted. ...In summary, by predicting a word at a position in a sentence by using a subset of its bidirectional context, and taking into account the word is itself not seen using the permutation scan rule above, we can avoid a word from seeing itself in a multilayered context. This scheme, however requires that we sample enough permutations so that we make full use of the bidirectional context. [http://towardsdatascience.com/xlnet-a-clever-language-modeling-solution-ab41e87798b0 XLNet — a clever language modeling solution | Ajit Rajasekharan - Medium]