ERNIE - Revision history

BPeat at 01:39, 18 May 2023

2023-05-18T01:39:51Z

BPeat at 20:08, 28 April 2023

2023-04-28T20:08:13Z

BPeat: Text replacement - "http:" to "https:"

2023-03-28T15:11:38Z

Text replacement - "http:" to "https:"

BPeat at 03:12, 9 February 2023

2023-02-09T03:12:06Z

BPeat at 14:27, 4 December 2020

2020-12-04T14:27:19Z

BPeat at 02:43, 27 December 2019

2019-12-27T02:43:30Z

BPeat at 02:14, 22 December 2019

2019-12-22T02:14:06Z

BPeat at 02:11, 22 December 2019

2019-12-22T02:11:59Z

BPeat at 02:09, 22 December 2019

2019-12-22T02:09:36Z

BPeat at 02:05, 22 December 2019

2019-12-22T02:05:32Z

← Older revision		Revision as of 01:39, 18 May 2023
Line 15:		Line 15:
	Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] and [[XLNet]] on 16 tasks including English tasks on GLUE benchmarks and several common tasks in [[Government Services#China\|Chinese]].		Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] and [[XLNet]] on 16 tasks including English tasks on GLUE benchmarks and several common tasks in [[Government Services#China\|Chinese]].

−	Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the [[Government Services#China\|Chinese]] language. In English, the word serves as the semantic unit—meaning a word pulled completely out of context still contains meaning. The same cannot be said for characters in [[Government Services#China\|Chinese]]. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in [[Government Services#China\|Chinese]] and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document. [https://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language \| Karen Hao - MIT Technology Review]	+	Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the [[Government Services#China\|Chinese]] language. In English, the word serves as the semantic unit—meaning a word pulled completely out of [[context]] still contains meaning. The same cannot be said for characters in [[Government Services#China\|Chinese]]. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in [[Government Services#China\|Chinese]] and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document. [https://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language \| Karen Hao - MIT Technology Review]

@@ Line 8: / Line 8: @@
 [https://www.google.com/search?q=ERNIE+natural+language ...Google search]
-* [[Natural Language Processing (NLP)]]
+* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]]  ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ...  [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
 * [https://research.baidu.com/Blog/index-view?id=121 Baidu’s Optimized ERNIE Achieves State-of-the-Art Results in Natural Language Processing Tasks | Baidu Research]
 * [https://arxiv.org/abs/1907.12412v1 ERNIE 2.0: A Continual Pre-training Framework for Language Understanding | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang]

@@ Line 5: / Line 5: @@
 |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
 }}
-[http://www.youtube.com/results?search_query=ERNIE+natural+language Youtube search...] |
+[https://www.youtube.com/results?search_query=ERNIE+natural+language Youtube search...] |
-[http://www.google.com/search?q=ERNIE+natural+language ...Google search]
+[https://www.google.com/search?q=ERNIE+natural+language ...Google search]
 * [[Natural Language Processing (NLP)]]
-* [http://research.baidu.com/Blog/index-view?id=121 Baidu’s Optimized ERNIE Achieves State-of-the-Art Results in Natural Language Processing Tasks | Baidu Research]
+* [https://research.baidu.com/Blog/index-view?id=121 Baidu’s Optimized ERNIE Achieves State-of-the-Art Results in Natural Language Processing Tasks | Baidu Research]
-* [http://arxiv.org/abs/1907.12412v1 ERNIE 2.0: A Continual Pre-training Framework for Language Understanding | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang]
+* [https://arxiv.org/abs/1907.12412v1 ERNIE 2.0: A Continual Pre-training Framework for Language Understanding | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang]
-* [http://github.com/PaddlePaddle/ERNIE ERNIE | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang - Baidu - GitHub]
+* [https://github.com/PaddlePaddle/ERNIE ERNIE | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang - Baidu - GitHub]
 Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] and [[XLNet]] on 16 tasks including English tasks on GLUE benchmarks and several common tasks in [[Government Services#China|Chinese]].
-Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the [[Government Services#China|Chinese]] language. In English, the word serves as the semantic unit—meaning a word pulled completely out of context still contains meaning. The same cannot be said for characters in [[Government Services#China|Chinese]]. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in [[Government Services#China|Chinese]] and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document.  [http://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language | Karen Hao - MIT Technology Review]
+Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the [[Government Services#China|Chinese]] language. In English, the word serves as the semantic unit—meaning a word pulled completely out of context still contains meaning. The same cannot be said for characters in [[Government Services#China|Chinese]]. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in [[Government Services#China|Chinese]] and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document.  [https://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language | Karen Hao - MIT Technology Review]
-<img src="http://github.com/PaddlePaddle/ERNIE/raw/develop/.metas/ernie2.0_arch.png" width="800" height="500">
+<img src="https://github.com/PaddlePaddle/ERNIE/raw/develop/.metas/ernie2.0_arch.png" width="800" height="500">
 <youtube>8K1IX7VJ5Fc</youtube>

@@ Line 2: / Line 2: @@
 |title=PRIMO.ai
 |titlemode=append
-|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Facebook
+|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, Facebook
 |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
 }}

@@ Line 2: / Line 2: @@
 |title=PRIMO.ai
 |titlemode=append
-|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, Tensorflow, Google, Nvidia, Microsoft, Azure, Amazon, AWS
+|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Facebook
 |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
 }}
@@ Line 13: / Line 13: @@
 * [http://github.com/PaddlePaddle/ERNIE ERNIE | Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang - Baidu - GitHub]
-Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] and [[XLNet]] on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese.
+Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms [[Bidirectional Encoder Representations from Transformers (BERT)]] and [[XLNet]] on 16 tasks including English tasks on GLUE benchmarks and several common tasks in [[Government Services#China|Chinese]].
-Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the Chinese language. In English, the word serves as the semantic unit—meaning a word pulled completely out of context still contains meaning. The same cannot be said for characters in Chinese. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in Chinese and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document.  [http://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language | Karen Hao - MIT Technology Review]
+Out of a full score of 100, the average person scores around 87 points. Baidu is now the first team to surpass 90 with its model, ERNIE. When Baidu researchers began developing their own language model, they wanted to build on the masking technique. But they realized they needed to tweak it to accommodate the [[Government Services#China|Chinese]] language. In English, the word serves as the semantic unit—meaning a word pulled completely out of context still contains meaning. The same cannot be said for characters in [[Government Services#China|Chinese]]. While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart. So the researchers trained ERNIE on a new version of masking that hides strings of characters rather than single ones. They also trained it to distinguish between meaningful and random strings so it could mask the right character combinations accordingly. As a result, ERNIE has a greater grasp of how words encode information in [[Government Services#China|Chinese]] and is much more accurate at predicting the missing pieces. This proves useful for applications like translation and information retrieval from a text document.  [http://www.technologyreview.com/s/614996/ai-baidu-ernie-google-bert-natural-language-glue/ Baidu has a new trick for teaching AI the meaning of language | Karen Hao - MIT Technology Review]

← Older revision		Revision as of 02:09, 22 December 2019
Line 9:		Line 9:

	* [[Natural Language Processing (NLP)]]		* [[Natural Language Processing (NLP)]]
		+	* [http://arxiv.org/abs/1907.12412v1 ERNIE 2.0: A Continual Pre-training Framework for Language Understanding \| Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang]
	* [http://github.com/PaddlePaddle/ERNIE ERNIE \| Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang - Baidu - GitHub]		* [http://github.com/PaddlePaddle/ERNIE ERNIE \| Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, H. Wang - Baidu - GitHub]