Difference between revisions of "Natural Language Processing (NLP)"

From
Jump to: navigation, search
(Capabilities)
(46 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools  
 
|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools  
 
}}
 
}}
[http://www.youtube.com/results?search_query=nlp+nli+natural+language+Processing Youtube search...] | [http://www.quora.com/topic/Natural-Language-Processing Quora search...]
+
[http://www.youtube.com/results?search_query=nlp+nli+natural+language+Processing Youtube search...] | [http://www.quora.com/topic/Natural-Language-Processing Quora search...]  
 
[http://www.google.com/search?q=nlp+nli+natural+language+Processing ...Google search]
 
[http://www.google.com/search?q=nlp+nli+natural+language+Processing ...Google search]
  
 
Speech recognition, (speech) translation, understanding (semantic parsing) complete sentences, understanding synonyms of matching words, sentiment analysis, and writing/generating complete grammatically correct sentences and paragraphs.
 
Speech recognition, (speech) translation, understanding (semantic parsing) complete sentences, understanding synonyms of matching words, sentiment analysis, and writing/generating complete grammatically correct sentences and paragraphs.
 
+
* [[Natural Language Processing (NLP) Techniques]]
 +
* [[Natural Language Tools & Services]]
 +
* Wikipedia:
 +
** [http://en.wikipedia.org/wiki/Outline_of_natural_language_processing#Natural_language_processing_tools Outline of Natural Language Processing]
 +
** [http://en.wikipedia.org/wiki/Natural_language_processing Natural Language Processing]
 +
** [http://en.wikipedia.org/wiki/Grammar_induction Grammar Induction]
 
* [http://newsletter.ruder.io/ NLP News | Sebastian Ruder]
 
* [http://newsletter.ruder.io/ NLP News | Sebastian Ruder]
* [[Natural Language Tools & Services]]
 
 
* [[Courses & Certifications#Natural Language Processing (NLP)|Natural Language Processing (NLP)]] Courses & Certifications
 
* [[Courses & Certifications#Natural Language Processing (NLP)|Natural Language Processing (NLP)]] Courses & Certifications
 
* [http://gluebenchmark.com/leaderboard NLP Models:]
 
* [http://gluebenchmark.com/leaderboard NLP Models:]
Line 22: Line 26:
 
** GPT-2 OpenAI… [http://towardsdatascience.com/too-powerful-nlp-model-generative-pre-training-2-4cc6afb6655 GPT-2 - Too powerful NLP model (GPT-2) | Edward Ma - Towards Data Science]
 
** GPT-2 OpenAI… [http://towardsdatascience.com/too-powerful-nlp-model-generative-pre-training-2-4cc6afb6655 GPT-2 - Too powerful NLP model (GPT-2) | Edward Ma - Towards Data Science]
 
** [[Attention]] Mechanism/[[Transformer]] Model
 
** [[Attention]] Mechanism/[[Transformer]] Model
** [[Recurrent Neural Network (RNN)]]
+
** Previous Efforts:
*** [[Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)]]
+
*** [[Autoencoders / Encoder-Decoders]]
* [http://dailynlp.com/nlp-techniques/ NLP Techniques: A Preliminary Cornerstone List | Martin Messier]
+
*** [[Gated Recurrent Unit (GRU)]]
* [[Text Transfer Learning]]
+
*** [[Deep Q Network (DQN)]]
* [http://en.wikipedia.org/wiki/Outline_of_natural_language_processing#Natural_language_processing_tools Outline of natural language processing | Wikipedia]
+
*** [[Recurrent Neural Network (RNN)]]
* [http://en.wikipedia.org/wiki/Natural_language_processing Natural Language Processing | Wikipedia]
+
*** [[Long Short-Term Memory (LSTM)]]
* [http://en.wikipedia.org/wiki/Grammar_induction Grammar Induction | Wikipedia]
+
**** [[Average-Stochastic Gradient Descent (SGD) Weight-Dropped LSTM (AWD-LSTM)]]
 +
* [http://pathmind.com/wiki/thought-vectors Thought Vectors | Geoffrey Hinton - A.I. Wiki - pathmind]
 +
* [http://medium.com/@datamonsters/artificial-neural-networks-for-natural-language-processing-part-1-64ca9ebfa3b2 7 types of Artificial Neural Networks for Natural Language Processing]
 +
 
 
* [http://www.quora.com/How-do-I-learn-Natural-Language-Processing How do I learn Natural Language Processing? | Sanket Gupta]  
 
* [http://www.quora.com/How-do-I-learn-Natural-Language-Processing How do I learn Natural Language Processing? | Sanket Gupta]  
 
* [http://www.chrisumbel.com/article/node_js_natural_language_nlp Natural Language | Chris Umbel]
 
* [http://www.chrisumbel.com/article/node_js_natural_language_nlp Natural Language | Chris Umbel]
 
* [http://azure.microsoft.com/en-us/services/cognitive-services/directory/lang/ Language services | Cognitive Services | Microsoft Azure]
 
* [http://azure.microsoft.com/en-us/services/cognitive-services/directory/lang/ Language services | Cognitive Services | Microsoft Azure]
 
* [http://www.tutorialspoint.com/natural_language_processing/natural_language_processing_quick_guide.htm Natural Language Processing - Quick Guide | TutorialsPoint]
 
* [http://www.tutorialspoint.com/natural_language_processing/natural_language_processing_quick_guide.htm Natural Language Processing - Quick Guide | TutorialsPoint]
* [[End-to-End Speech]]
+
* [http://web.stanford.edu/~jurafsky/slp3/ Speech and Language Processing | Dan Jurafsky and James H. Martin] (3rd ed. draft)
 +
* [http://nlp.stanford.edu/projects/snli/ The Stanford Natural Language Inference (SNLI) Corpus]
 +
* NLP/NLU/NLI [[Benchmarks]]:
 +
** [[Benchmarks#GLUE|General Language Understanding Evaluation (GLUE)]]
 +
** [[Benchmarks#SQuAD|The Stanford Question Answering Dataset (SQuAD)]]
 +
** [[Benchmarks#RACE|ReAding Comprehension (RACE)]]
 +
 
 +
 
 +
Over the last two years, the Natural Language Processing community has witnessed an acceleration in progress on a wide range of different tasks and applications. This progress was enabled by a shift of paradigm in the way we classically build an NLP system  [http://medium.com/huggingface/the-best-and-most-current-of-modern-natural-language-processing-5055f409a1d1 The Best and Most Current of Modern Natural Language Processing | Victor Sanh - Medium]:
 +
* For a long time, we used pre-trained word embeddings such as [[Word2Vec]] or [[Global Vectors for Word Representation (GloVe)]] to initialize the first layer of a neural network, followed by a task-specific architecture that is trained in a supervised way using a single dataset.
 +
* Recently, several works demonstrated that we can learn hierarchical contextualized representations on web-scale datasets leveraging unsupervised (or self-supervised) signals such as language modeling and transfer this pre-training to downstream tasks ([[Transfer Learning]]). Excitingly, this shift led to significant advances on a wide range of downstream applications ranging from Question Answering, to Natural Language Inference through Syntactic Parsing…
  
  
Line 44: Line 61:
 
* [[Language Translation]]
 
* [[Language Translation]]
 
* [[Assistants]] - Dialog Systems
 
* [[Assistants]] - Dialog Systems
 +
* [[Summarization / Paraphrasing]]
 +
* [[Sentiment Analysis]]
 +
* [[Wikifier]]
 
* [[Natural Language Generation (NLG)]]  ...Text Analytics
 
* [[Natural Language Generation (NLG)]]  ...Text Analytics
  
 
<img src="http://www.e-spirit.com/images/Blog/2018/07_NLG_future_of_content_management/NLG_Future_of_Content_Management_Content_01.png" width="900" height="450">
 
<img src="http://www.e-spirit.com/images/Blog/2018/07_NLG_future_of_content_management/NLG_Future_of_Content_Management_Content_01.png" width="900" height="450">
  
== <span id="Summarization / Paraphrasing"></span>Summarization / Paraphrasing ==
+
= <span id="Natural Language Understanding (NLU)"></span>Natural Language Understanding (NLU) =
 +
[http://www.youtube.com/results?search_query=Understanding+NLU+natural+language+inference+entailment+RTE+Text+Speech Youtube search...] | [http://www.quora.com/topic/Natural-Language-Processing Quora search...]
 +
[http://www.google.com/search?q=Understanding+NLU+nli+natural+language+inference+entailment+RTE+Text+Speech ...Google search]
  
[http://www.youtube.com/results?search_query=Summarizer+nlp+natural+language Youtube search...]
+
* [[Natural Language Processing (NLP)#Phonology (Phonetics)|Phonology (Phonetics)]]
[http://www.google.com/search?q=Summarizer+nlp+natural+language ...Google search]
+
* [[Natural Language Processing (NLP)#Lexical (Morphology)|Lexical (Morphology)]]
 +
* [[Natural Language Processing (NLP)#Syntax (Parsing)| Syntax (Parsing)]]
 +
* [[Natural Language Processing (NLP)#Semantics| Semantics]]
 +
* [[Natural Language Processing (NLP)#Discourse (Dialog)| Discourse (Dialog)]]
 +
* [[Natural Language Processing (NLP)#Pragmatics| Pragmatics]]
  
* [http://techxplore.com/news/2019-04-neural-network-scientific-papers-plain-english.html A neural network can read scientific papers and render a plain-English summary | David L. Chandler - TechExplore]
+
Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis.
* [[Data Augmentation#Auto-tagging|Auto-tagging]]
+
NLU is the post-processing of text, after the use of NLP algorithms (identifying parts-of-speech, etc.), that utilizes context from recognition devices (automatic speech recognition [ASR], vision recognition, last conversation, misrecognized words from ASR, personalized profiles, microphone proximity etc.), in all of its forms, to discern meaning of fragmented and run-on sentences to execute an intent from typically voice commands. NLU has an ontology around the particular product vertical that is used to figure out the probability of some intent. An NLU has a defined list of known intents that derives the message payload from designated contextual information recognition sources. The NLU will provide back multiple message outputs to separate services (software) or resources (hardware) from a single derived intent (response to voice command initiator with visual sentence (shown or spoken) and transformed voice command message too different output messages to be consumed for M2M communications and actions)  [http://en.wikipedia.org/wiki/Natural-language_understanding Natural-language understanding | Wikipedia]  
  
<youtube>ogrJaOIuBx4</youtube>
+
NLU uses algorithms to reduce human speech into a structured ontology. Then AI algorithms detect such things as intent, timing, locations, and sentiments. ... Natural language understanding is the first step in many processes, such as categorizing text, gathering news, archiving individual pieces of text, and, on a larger scale, analyzing content. Real-world examples of NLU range from small tasks like issuing short commands based on comprehending text to some small degree, like rerouting an email to the right person based on basic syntax and a decently-sized lexicon. Much more complex endeavors might be fully comprehending news articles or shades of meaning within poetry or novels. [http://www.kdnuggets.com/2019/07/nlp-vs-nlu-understanding-language-processing.html NLP vs. NLU: from Understanding a Language to Its Processing | Sciforce]
<youtube>0DLa5sApXhc</youtube>
 
  
 +
http://cdn-images-1.medium.com/max/1000/1*Uf_qQ0zF8G8y9zUhndA08w.png
  
== <span id="Sentiment Analysis"></span>Sentiment Analysis ==
+
Image Source: [http://nlp.stanford.edu/~wcmac/papers/20140716-UNLU.pdf Understanding Natural Language Understanding | Bill MacCartney]
  
[http://www.youtube.com/results?search_query=Sentiment+Analysis+nlp+natural+language Youtube search...]
+
<youtube>mhHfnhh-pB4</youtube>
[http://www.google.com/search?q=Sentiment+Analysis+nlp+natural+language ...Google search]
+
<youtube>1-8KJV8aSQM</youtube>
 +
<youtube>ycXWAtm22-w</youtube>
  
* [http://web.stanford.edu/class/cs124/lec/sentiment2017.pdf Sentiment Analysis | Stanford]
+
== <span id="Phonology (Phonetics)"></span>[http://en.wikipedia.org/wiki/Phonology Phonology (Phonetics)] ==
* [[Data Augmentation#Auto-tagging|Auto-tagging]]
+
[http://www.youtube.com/results?search_query=phonetics+phoneme+nlp+natural+language Youtube search...]
* [[Watch me Build a Finance Startup]] | Siraj Raval 
+
[http://www.google.com/search?q=phonetics+phoneme+nlp+natural+language ...Google search]
  
Sentiment Analysis algorithms work by referring external resources where the positive and negative polarity of each word is considered. These external words, for which the polarity is predetermined, are known as lexicons. There are several lists of lexicons available and each one focuses on the polarity of a given word in a particular context. [http://subscription.packtpub.com/book/big_data_and_business_intelligence/9781783989348/6/ch06lvl1sec57/a-baseline-algorithm-for-sa-using-sentiwordnet-lexicons A baseline algorithm for SA using SentiWordNet lexicons Packt]
+
Phonology is a branch of linguistics concerned with the systematic organization of sounds in spoken languages and signs in sign languages. It used to be only the study of the systems of phonemes in spoken languages (and therefore used to be also called phonemics, or phonematics), but it may also cover any linguistic analysis either at a level beneath the word (including syllable, onset and rime, articulatory gestures, articulatory features, mora, etc.) or at all levels of language where sound or signs are structured to convey linguistic meaning.
  
<youtube>AJVP96tAWxw</youtube>
+
A <b>Phoneme</b> is the most basic sound unit of sound; any of the perceptually distinct units of sound in a specified language that distinguish one word from another, for example p, b, d, and t in the English words pad, pat, bad, and bat. [http://en.wikipedia.org/wiki/Phoneme Phoneme | Wikipedia]
<youtube>eObouMO2qSE</youtube>
 
<youtube>SB8ckgT8l9c</youtube>
 
<youtube>ojDHK1SmCHA</youtube>
 
 
 
 
 
=== Baseline Algorithms ===
 
* [http://gab41.lab41.org/a-tour-of-sentiment-analysis-techniques-getting-a-baseline-for-sunny-side-up-45730ec03330#.kvxpv0cct A Tour of Sentiment Analysis Techniques: Getting a Baseline for Sunny Side Up | Vishal S. -] [http://gab41.lab41.org/ Gab41]
 
* [http://www.aclweb.org/anthology/L16-1427/ Reliable Baselines for Sentiment Analysis in Resource-Limited Languages: The Serbian Movie Review Dataset | V. Batanović, B. Nikolić, and M. Milosavljević]
 
* [http://www.lexalytics.com/lexablog/sentiment-accuracy-baseline-testing Sentiment Accuracy: Explaining the Baseline and How to Test It | Paul Barba -  Lexalytics]
 
 
 
<youtube>tAkgHtgu3D0</youtube>
 
 
 
=== Sentiment Lexicon ===
 
 
 
* [http://www.cfilt.iitb.ac.in/resources/surveys/Sentiment-Lexicons-Sagar-Ahire.pdf A Survey of Sentiment Lexicons | Sagar Ahire]
 
* [http://github.com/aesuli/sentiwordnet SentiWordNet - GitHub] a lexical resource for opinion mining. SentiWordNet assigns to each synset of WordNet three sentiment scores: positivity, negativity, objectivity
 
* [http://nlp.stanford.edu/sentiment/treebank.html Sentiment Treebank | Stanford]
 
* [http://github.com/sfu-discourse-lab/SO-CAL SO-CAL] is the Semantic Orientation CALculator, a tool to extract sentiment from text. Sentiment is defined as positive or negative opinion.
 
 
 
- a database of lexical units for a language along with their sentiment orientations. This can be expressed as a set of tuples of the form (lexical unit, sentiment). Here, the lexical units may be words, word senses, phrases, etc.
 
 
 
<youtube>wBE0FE_2ddE</youtube>
 
 
 
== <span id="Wikifier"></span>Wikifier ==
 
[http://www.youtube.com/results?search_query=Wikifier+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=Wikifier+nlp+natural+language ...Google search]
 
 
 
* [http://cogcomp.org/page/software_view/Wikifier Illinois Wikifier]
 
* [http://wikifier.org/ Wikifier Demo - Semantic Annotation Service for 100 Languages | Wikifier.org]
 
** [http://wikifier.org/info.html The JSI Wikifier is a web service that takes a text document as input and annotates it with links to relevant Wikipedia concepts.]
 
 
 
<youtube>IkryLTdogjw</youtube>
 
  
 +
A <b>Grapheme</b> is the smallest unit of a writing system of any given language. An individual grapheme may or may not carry meaning by itself, and may or may not correspond to a single phoneme of the spoken language
  
 +
<youtube>IKS2DQ_6RHQ</youtube>
 +
<youtube>OlCtIqsqvrY</youtube>
  
<hr>
 
  
<img src="http://miro.medium.com/max/702/1*Qtk5pN8n_BcYUsosrKFrFg.png" width="400" height="200">  [http://towardsdatascience.com/linguistic-knowledge-in-natural-language-processing-332630f43ce1 Linguistic Knowledge in Natural Language Processing | Hafidz Zulkifli | Towards Data Science - Medium]
+
== <span id="Lexical (Morphology)"></span>[http://en.wikipedia.org/wiki/Morphology_(linguistics) Lexical (Morphology)] ==
 
 
=<span id="Techniques"></span>Techniques =
 
 
 
<hr>
 
 
 
== <span id="Morphology"></span>[http://en.wikipedia.org/wiki/Morphology_(linguistics) Morphology] ==
 
 
<b>Lexical Ambiguity</b> – Words have multiple meanings
 
<b>Lexical Ambiguity</b> – Words have multiple meanings
  
Line 255: Line 246:
  
  
<img src="http://miro.medium.com/max/702/1*Qtk5pN8n_BcYUsosrKFrFg.png" width="400" height="200">
 
  
 
<hr>
 
<hr>
Line 394: Line 384:
 
<youtube>EymPQgCtcAE</youtube>
 
<youtube>EymPQgCtcAE</youtube>
  
<img src="http://miro.medium.com/max/702/1*Qtk5pN8n_BcYUsosrKFrFg.png" width="400" height="200">
+
 
  
 
<hr>
 
<hr>
Line 437: Line 427:
 
[http://www.google.com/search?q=Named+Entity+Recognition+NER+nlp+natural+language ...Google search]
 
[http://www.google.com/search?q=Named+Entity+Recognition+NER+nlp+natural+language ...Google search]
  
 +
* [[NLP Keras model in browser with TensorFlow.js]]
 
* [http://medium.com/explore-artificial-intelligence/introduction-to-named-entity-recognition-eda8c97c2db1 Introduction to Named Entity Recognition | Suvro Banerjee - Medium]
 
* [http://medium.com/explore-artificial-intelligence/introduction-to-named-entity-recognition-eda8c97c2db1 Introduction to Named Entity Recognition | Suvro Banerjee - Medium]
 
* [http://stanfordnlp.github.io/CoreNLP/ner.html NERClassifierCombiner | Stanford CoreNLP]
 
* [http://stanfordnlp.github.io/CoreNLP/ner.html NERClassifierCombiner | Stanford CoreNLP]
* [[NLP Keras model in browser with TensorFlow.js]]
+
* [http://www.kdnuggets.com/2018/08/named-entity-recognition-practitioners-guide-nlp-4.html Named Entity Recognition: A Practitioner’s Guide to NLP | Dipanjan Sarkar - RedHat KDnuggets]
  
Named Entities (also known as entity identification, entity chunking, sequence tagging, [[Natural Language Processing (NLP)#Part-of-Speech (POS) Tagging|Part-of-Speech (POS) Tagging]], and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.  Most research on NER systems has been structured as taking an unannotated block of text, and producing an annotated block of text that highlights the names of entities.
+
Named Entities (also known as entity identification, entity chunking, sequence tagging, [[Natural Language Processing (NLP)#Part-of-Speech (POS) Tagging|Part-of-Speech (POS) Tagging]], and entity chunking/extraction) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc.  Most research on NER systems has been structured as taking an unannotated block of text, and producing an annotated block of text that highlights the names of entities.
  
 
http://joecyw.files.wordpress.com/2017/02/maxthonsnap20170219114627.png
 
http://joecyw.files.wordpress.com/2017/02/maxthonsnap20170219114627.png
Line 476: Line 467:
  
  
<img src="http://miro.medium.com/max/702/1*Qtk5pN8n_BcYUsosrKFrFg.png" width="400" height="200">
+
<hr>
 +
== <span id="Discourse (Dialog)"></span>[http://en.wikipedia.org/wiki/Discourse_analysis Discourse (Dialog)] ==
 +
Discourse is the creation and organization of the segments of a language above as well as below the sentence. It is segments of language which may be bigger or smaller than a single sentence but the adduced meaning is always beyond the sentence.
 +
 
 +
* identifying the discourse structure of a connected text, i.e. the nature of the discourse relationships between sentences (e.g. elaboration, explanation, contrast).
 +
* recognizing and classifying the speech acts in a chunk of text (e.g. yes-no question, content question, statement, assertion, etc.).
 +
 
 +
The objects of discourse analysis (discourse, writing, conversation, communicative event) are variously defined in terms of coherent sequences of sentences, propositions, speech, or turns-at-talk.
 +
 
 +
The problems addressed in discourse research aim to answer two general kinds of questions:
 +
# what information is contained in extended sequences of utterances that goes beyond the meaning of the individual utterances themselves?
 +
# how does the context in which an utterance is used a�ect the meaning of the individual utterances, or parts of them?
 +
 
 +
[http://www.ccs.neu.edu/home/futrelle/bionlp/hlt/chpt6.pdf Chapter 6 Discourse and Dialogue]
 +
 
 +
 
 +
 
 +
 
 +
<b> Compared to traditional linguistics </b>
 +
Contrary to much of traditional linguistics, discourse analysts not only study language use 'beyond the sentence boundary' but also prefer to analyze 'naturally occurring' language use, not invented examples. Text linguistics is a closely related field. The essential difference between discourse analysis and text linguistics is that discourse analysis aims at revealing socio-psychological characteristics of a person/persons rather than text structure.
 +
 
  
 
<hr>
 
<hr>
Line 567: Line 578:
 
* [http://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html BERT: State of the Art NLP Model, Explained - Rani Horev - KDnuggets]
 
* [http://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html BERT: State of the Art NLP Model, Explained - Rani Horev - KDnuggets]
  
Training the language model in BERT is done by predicting 15% of the tokens in the input, that were randomly picked. These tokens are pre-processed as follows — 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The intuition that led the authors to pick this approach is as follows (Thanks to [http://ai.google/research/people/106320 Jacob Devlin] from Google for the insight):
+
Training the language model in [[Bidirectional Encoder Representations from Transformers (BERT)]] is done by predicting 15% of the tokens in the input, that were randomly picked. These tokens are pre-processed as follows — 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The intuition that led the authors to pick this approach is as follows (Thanks to [http://ai.google/research/people/106320 Jacob Devlin] from Google for the insight):
  
 
* If we used [MASK] 100% of the time the model wouldn’t necessarily produce good token representations for non-masked words. The non-masked tokens were still used for context, but the model was optimized for predicting masked words.
 
* If we used [MASK] 100% of the time the model wouldn’t necessarily produce good token representations for non-masked words. The non-masked tokens were still used for context, but the model was optimized for predicting masked words.
Line 573: Line 584:
 
* If we used [MASK] 90% of the time and kept the same word 10% of the time, then the model could just trivially copy the non-contextual embedding.
 
* If we used [MASK] 90% of the time and kept the same word 10% of the time, then the model could just trivially copy the non-contextual embedding.
  
=== <span id="Natural Language Understanding (NLU)"></span>Natural Language Understanding (NLU) ===
+
<youtube>Fs8Zb4T-_CE</youtube>
[http://www.youtube.com/results?search_query=Understanding+NLU+natural+language+inference+entailment+RTE+Text+Speech Youtube search...] | [http://www.quora.com/topic/Natural-Language-Processing Quora search...]
 
[http://www.google.com/search?q=Understanding+NLU+nli+natural+language+inference+entailment+RTE+Text+Speech ...Google search]
 
 
 
* [http://web.stanford.edu/~jurafsky/slp3/ Speech and Language Processing | Dan Jurafsky and James H. Martin] (3rd ed. draft)
 
* [http://nlp.stanford.edu/projects/snli/ The Stanford Natural Language Inference (SNLI) Corpus]
 
* [http://www.nltk.org/book/ch07.html Extracting Information from Text | NLTK.org]
 
* [[Topic Model/Mapping]]
 
* [http://techxplore.com/news/2019-02-neural-networks-microscope.html Linguistic Correlation Analysis - Putting neural networks under the microscope | Rob Matheson]
 
* [http://www.engadget.com/2018/07/24/google-docs-ai-grammar-checking/ Google Docs uses AI to catch your grammar mistakes | Jon Fingas - Engadget]
 
* [[Document Understanding AI]] | Google
 
* [[Sequence to Sequence (Seq2Seq)]]
 
* NLP/NLU/NLI [[Benchmarks]]:
 
** [[Benchmarks#GLUE|General Language Understanding Evaluation (GLUE)]]
 
** [[Benchmarks#SQuAD|The Stanford Question Answering Dataset (SQuAD)]]
 
** [[Benchmarks#RACE|ReAding Comprehension (RACE)]]
 
 
 
  
Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis.
+
=== <span id="Managed Vocabularies"></span>Managed Vocabularies ===
 
 
NLU is the post-processing of text, after the use of NLP algorithms (identifying parts-of-speech, etc.), that utilizes context from recognition devices (automatic speech recognition [ASR], vision recognition, last conversation, misrecognized words from ASR, personalized profiles, microphone proximity etc.), in all of its forms, to discern meaning of fragmented and run-on sentences to execute an intent from typically voice commands. NLU has an ontology around the particular product vertical that is used to figure out the probability of some intent. An NLU has a defined list of known intents that derives the message payload from designated contextual information recognition sources. The NLU will provide back multiple message outputs to separate services (software) or resources (hardware) from a single derived intent (response to voice command initiator with visual sentence (shown or spoken) and transformed voice command message too different output messages to be consumed for M2M communications and actions)  [http://en.wikipedia.org/wiki/Natural-language_understanding Natural-language understanding | Wikipedia]
 
 
 
NLU uses algorithms to reduce human speech into a structured ontology. Then AI algorithms detect such things as intent, timing, locations, and sentiments. ... Natural language understanding is the first step in many processes, such as categorizing text, gathering news, archiving individual pieces of text, and, on a larger scale, analyzing content. Real-world examples of NLU range from small tasks like issuing short commands based on comprehending text to some small degree, like rerouting an email to the right person based on basic syntax and a decently-sized lexicon. Much more complex endeavors might be fully comprehending news articles or shades of meaning within poetry or novels. [http://www.kdnuggets.com/2019/07/nlp-vs-nlu-understanding-language-processing.html NLP vs. NLU: from Understanding a Language to Its Processing | Sciforce]
 
 
 
http://cdn-images-1.medium.com/max/1000/1*Uf_qQ0zF8G8y9zUhndA08w.png
 
 
 
Image Source: [http://nlp.stanford.edu/~wcmac/papers/20140716-UNLU.pdf Understanding Natural Language Understanding | Bill MacCartney]
 
 
 
<youtube>mhHfnhh-pB4</youtube>
 
<youtube>1-8KJV8aSQM</youtube>
 
 
 
<hr>
 
 
 
== <span id="Managed Vocabularies"></span>Managed Vocabularies ==
 
  
 
[[Datasets]]
 
[[Datasets]]
  
=== <span id="Corpora"></span>[http://en.wikipedia.org/wiki/Text_corpus Corpora] ===
+
==== <span id="Corpora"></span>[http://en.wikipedia.org/wiki/Text_corpus Corpora] ====
 
[http://www.youtube.com/results?search_query=Corpora+nlp+natural+language Youtube search...]
 
[http://www.youtube.com/results?search_query=Corpora+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=Corpora+nlp+natural+language ...Google search]
 
[http://www.google.com/search?q=Corpora+nlp+natural+language ...Google search]
Line 622: Line 602:
 
<youtube>T68P5-8tM-Y</youtube>
 
<youtube>T68P5-8tM-Y</youtube>
  
=== <span id="Ontology"></span>[http://en.wikipedia.org/wiki/Ontology Ontology] ===
+
==== <span id="Ontology"></span>[http://en.wikipedia.org/wiki/Ontology Ontology] ====
 
[http://www.youtube.com/results?search_query=Ontologies+Ontology+nlp+natural+language Youtube search...]
 
[http://www.youtube.com/results?search_query=Ontologies+Ontology+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=Ontologies+Ontology+nlp+natural+language ...Google search]
 
[http://www.google.com/search?q=Ontologies+Ontology+nlp+natural+language ...Google search]
Line 642: Line 622:
 
<youtube>dVZFlg1pN8A</youtube>
 
<youtube>dVZFlg1pN8A</youtube>
  
==== <span id="Taxonomy"></span>[http://en.wikipedia.org/wiki/Taxonomy Taxonomy] ====
+
===== <span id="Taxonomy"></span>[http://en.wikipedia.org/wiki/Taxonomy Taxonomy] =====
 
[http://www.youtube.com/results?search_query=Taxonomies+Taxonomy+nlp+natural+language Youtube search...]
 
[http://www.youtube.com/results?search_query=Taxonomies+Taxonomy+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=Taxonomies+taxonomy+nlp+natural+language ...Google search]
 
[http://www.google.com/search?q=Taxonomies+taxonomy+nlp+natural+language ...Google search]
Line 655: Line 635:
 
<youtube>6VI0QAzz6LM</youtube>
 
<youtube>6VI0QAzz6LM</youtube>
  
<hr>
+
= Related =
 
 
== <span id="Deep Learning Algorithms"></span>Deep Learning Algorithms ==
 
[http://www.youtube.com/results?search_query=deep+learning+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=deep+learning+nlp+natural+language ...Google search]
 
 
 
* [[Deep Q Network (DQN)]]
 
* [http://medium.com/@datamonsters/artificial-neural-networks-for-natural-language-processing-part-1-64ca9ebfa3b2 7 types of Artificial Neural Networks for Natural Language Processing]
 
* [http://www.aclweb.org/anthology/C16-1229 Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts | Xingyou Wang, Weijie Jiang, Zhiyong Luo]
 
* [[Autoencoders / Encoder-Decoders]]
 
* [[LSTM]]
 
* [[GRU]]
 
 
 
 
 
<youtube>p5K_xtldcp4</youtube>
 
<youtube>DDByc9LyMV8</youtube>
 
<youtube>DDByc9LyMV8</youtube>
 
 
 
 
 
<hr>
 
  
 
== <span id="Workbench / Pipeline"></span>Workbench / Pipeline ==
 
== <span id="Workbench / Pipeline"></span>Workbench / Pipeline ==
Line 680: Line 641:
 
[http://www.google.com/search?q=Pipeline+workflow+workbench+nlp+natural+language ...Google search]
 
[http://www.google.com/search?q=Pipeline+workflow+workbench+nlp+natural+language ...Google search]
  
 +
* [[Pipeline]] e.g. Google [[Kubeflow Pipelines]]
 
* [[Natural Language Tools & Services]]   
 
* [[Natural Language Tools & Services]]   
* [http://en.wikipedia.org/wiki/Outline_of_natural_language_processing#Natural_language_processing_tools Outline of natural language processing | Wikipedia]
 
 
* [http://github.com/CogComp/cogcomp-nlp/tree/master/pipeline CogComp NLP Pipeline | Cognitive Computation Group, led by Prof. Dan Roth]
 
* [http://github.com/CogComp/cogcomp-nlp/tree/master/pipeline CogComp NLP Pipeline | Cognitive Computation Group, led by Prof. Dan Roth]
 
* [http://azure.github.io/LearnAI-CustomTextAnalyticsWithAML/ Building Custom Text Analytics solutions using Azure Machine Learning (AML)]  
 
* [http://azure.github.io/LearnAI-CustomTextAnalyticsWithAML/ Building Custom Text Analytics solutions using Azure Machine Learning (AML)]  
 
* [[H2O]] Driverless AI
 
* [[H2O]] Driverless AI
 
* [http://gate.ac.uk/ GATE] toolkit
 
* [http://gate.ac.uk/ GATE] toolkit
* [[Pipeline]] e.g. Google [[Kubeflow Pipelines]]
 
 
* [http://uima.apache.org/ Apache UIMA] - Unstructured Information Management Applications
 
* [http://uima.apache.org/ Apache UIMA] - Unstructured Information Management Applications
 
* [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5486409/ An Ontology-Enabled Natural Language Processing Pipeline for Provenance Metadata Extraction from Biomedical Text | J. Valdez, M. Rueschman, M. Kim, S. Redline, and S. Sahoo]
 
* [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5486409/ An Ontology-Enabled Natural Language Processing Pipeline for Provenance Metadata Extraction from Biomedical Text | J. Valdez, M. Rueschman, M. Kim, S. Redline, and S. Sahoo]
Line 698: Line 658:
 
<youtube>IM99o2B_0Nw</youtube>
 
<youtube>IM99o2B_0Nw</youtube>
 
<youtube>k5X12mdEvb8</youtube>
 
<youtube>k5X12mdEvb8</youtube>
 
= Related =
 
== <span id="Phonetics"></span>Phonetics ==
 
[http://www.youtube.com/results?search_query=phonetics+phoneme+nlp+natural+language Youtube search...]
 
[http://www.google.com/search?q=phonetics+phoneme+nlp+natural+language ...Google search]
 
 
A <b>Phoneme</b> is the most basic sound unit of sound; any of the perceptually distinct units of sound in a specified language that distinguish one word from another, for example p, b, d, and t in the English words pad, pat, bad, and bat. [http://en.wikipedia.org/wiki/Phoneme Phoneme | Wikipedia]
 
 
A <b>Grapheme</b> is the smallest unit of a writing system of any given language. An individual grapheme may or may not carry meaning by itself, and may or may not correspond to a single phoneme of the spoken language
 
 
<youtube>IKS2DQ_6RHQ</youtube>
 
<youtube>OlCtIqsqvrY</youtube>
 
  
 
== Sapir Whorf Hypothesis - How language shapes the way we think ==
 
== Sapir Whorf Hypothesis - How language shapes the way we think ==
Line 718: Line 666:
 
<youtube>QglKeIIC5Ds</youtube>
 
<youtube>QglKeIIC5Ds</youtube>
  
== General Videos ==
+
== General Natural Language Processing (NLP) Videos ==
  
 
<youtube>OQQ-W_63UgQ</youtube>
 
<youtube>OQQ-W_63UgQ</youtube>
Line 726: Line 674:
 
<youtube>GazFsfcijXQ</youtube>
 
<youtube>GazFsfcijXQ</youtube>
 
<youtube>bDxFvr1gpSU</youtube>
 
<youtube>bDxFvr1gpSU</youtube>
 +
<youtube>p5K_xtldcp4</youtube>
 +
<youtube>DDByc9LyMV8</youtube>
 +
<youtube>DDByc9LyMV8</youtube>

Revision as of 22:08, 3 May 2020

Youtube search... | Quora search... ...Google search

Speech recognition, (speech) translation, understanding (semantic parsing) complete sentences, understanding synonyms of matching words, sentiment analysis, and writing/generating complete grammatically correct sentences and paragraphs.


Over the last two years, the Natural Language Processing community has witnessed an acceleration in progress on a wide range of different tasks and applications. This progress was enabled by a shift of paradigm in the way we classically build an NLP system The Best and Most Current of Modern Natural Language Processing | Victor Sanh - Medium:

  • For a long time, we used pre-trained word embeddings such as Word2Vec or Global Vectors for Word Representation (GloVe) to initialize the first layer of a neural network, followed by a task-specific architecture that is trained in a supervised way using a single dataset.
  • Recently, several works demonstrated that we can learn hierarchical contextualized representations on web-scale datasets leveraging unsupervised (or self-supervised) signals such as language modeling and transfer this pre-training to downstream tasks (Transfer Learning). Excitingly, this shift led to significant advances on a wide range of downstream applications ranging from Question Answering, to Natural Language Inference through Syntactic Parsing…


future-applications-of-nlp.png Artificial Intelligence Overview and Applications | Jagreet Kaur Gill

Capabilities

Natural Language Understanding (NLU)

Youtube search... | Quora search... ...Google search

Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis. NLU is the post-processing of text, after the use of NLP algorithms (identifying parts-of-speech, etc.), that utilizes context from recognition devices (automatic speech recognition [ASR], vision recognition, last conversation, misrecognized words from ASR, personalized profiles, microphone proximity etc.), in all of its forms, to discern meaning of fragmented and run-on sentences to execute an intent from typically voice commands. NLU has an ontology around the particular product vertical that is used to figure out the probability of some intent. An NLU has a defined list of known intents that derives the message payload from designated contextual information recognition sources. The NLU will provide back multiple message outputs to separate services (software) or resources (hardware) from a single derived intent (response to voice command initiator with visual sentence (shown or spoken) and transformed voice command message too different output messages to be consumed for M2M communications and actions) Natural-language understanding | Wikipedia

NLU uses algorithms to reduce human speech into a structured ontology. Then AI algorithms detect such things as intent, timing, locations, and sentiments. ... Natural language understanding is the first step in many processes, such as categorizing text, gathering news, archiving individual pieces of text, and, on a larger scale, analyzing content. Real-world examples of NLU range from small tasks like issuing short commands based on comprehending text to some small degree, like rerouting an email to the right person based on basic syntax and a decently-sized lexicon. Much more complex endeavors might be fully comprehending news articles or shades of meaning within poetry or novels. NLP vs. NLU: from Understanding a Language to Its Processing | Sciforce

1*Uf_qQ0zF8G8y9zUhndA08w.png

Image Source: Understanding Natural Language Understanding | Bill MacCartney

Phonology (Phonetics)

Youtube search... ...Google search

Phonology is a branch of linguistics concerned with the systematic organization of sounds in spoken languages and signs in sign languages. It used to be only the study of the systems of phonemes in spoken languages (and therefore used to be also called phonemics, or phonematics), but it may also cover any linguistic analysis either at a level beneath the word (including syllable, onset and rime, articulatory gestures, articulatory features, mora, etc.) or at all levels of language where sound or signs are structured to convey linguistic meaning.

A Phoneme is the most basic sound unit of sound; any of the perceptually distinct units of sound in a specified language that distinguish one word from another, for example p, b, d, and t in the English words pad, pat, bad, and bat. Phoneme | Wikipedia

A Grapheme is the smallest unit of a writing system of any given language. An individual grapheme may or may not carry meaning by itself, and may or may not correspond to a single phoneme of the spoken language


Lexical (Morphology)

Lexical Ambiguity – Words have multiple meanings

The study of words, how they are formed, and their relationship to other words in the same language. It analyzes the structure of words and parts of words, such as stems, root words, prefixes, and suffixes. Morphology also looks at parts of speech, intonation and stress, and the ways context can change a word's pronunciation and meaning.About the words that make up the sentence, how they are formed, and how do they change depending on their context. Some examples of these include:

  • Prefixes/suffixes
  • Singularization/pluralization
  • Gender detection
  • Word inflection (modification of word to express different grammatical categories such tenses, case, voice etc..). Other forms of inflection includes conjugation (inflection of verbs) and declension (inflection of nouns, adjectives, adverbs etc…).
  • Lemmatization (the base form of the word, or the reverse of inflection)
  • Spell checking

Text Preprocessing

Cleaning and preparation the information for use, such as punctuation removal, spelling correction, lowercasing, stripping markup tags (HTML,XML)

Regular Expressions (Regex)

Youtube search... ...Google search

Search for text patterns, validate emails and URLs, capture information, and use patterns to save development time.

regex-example.png

Soundex

Youtube search... ...Google search

a phonetic algorithm for indexing names by sound, as pronounced in English. The goal is for homophones to be encoded to the same representation so that they can be matched despite minor differences in spelling. The Soundex code for a name consists of a letter followed by three numerical digits: the letter is the first letter of the name, and the digits encode the remaining consonants. Consonants at a similar place of articulation share the same digit so, for example, the labial consonants B, F, P, and V are each encoded as the number 1. Wikipedia

The correct value can be found as follows:

  1. Retain the first letter of the name and drop all other occurrences of a, e, i, o, u, y, h, w.
  2. Replace consonants with digits as follows (after the first letter):
    • b, f, p, v → 1
    • c, g, j, k, q, s, x, z → 2
    • d, t → 3
    • l → 4
    • m, n → 5
    • r → 6
  3. If two or more letters with the same number are adjacent in the original name (before step 1), only retain the first letter; also two letters with the same number separated by 'h' or 'w' are coded as a single number, whereas such letters separated by a vowel are coded twice. This rule also applies to the first letter.
  4. If you have too few letters in your word that you can't assign three numbers, append with zeros until there are three numbers. If you have more than 3 letters, just retain the first 3 numbers.


Tokenization / Sentence Splitting

Youtube search... ...Google search

Tokenization is the process of demarcating (breaking text into individual words) and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input. A token (or n-gram) is a contiguous sequence of n items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application.

In-this-time-series-Google-Ngram-Viewer-is-used-to-compare-some-literature-for-children.png

Normalization

Youtube search... ...Google search

Process that converts a list of words to a more uniform sequence. .

Stemming (Morphological Similarity)

Youtube search... ...Google search

Stemmers remove morphological affixes from words, leaving only the word stem. Refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.

figure3.png

Lemmatization

Youtube search... ...Google search

Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . If confronted with the token saw, stemming might return just s, whereas lemmatization would attempt to return either see or saw depending on whether the use of the token was as a verb or a noun. The two may also differ in that stemming most commonly collapses derivationally related words, whereas lemmatization commonly only collapses the different inflectional forms of a lemma. Stemming and lemmatization | Stanford.edu NLTK's lemmatizer knows "am" and "are" are related to "be."

Capitalization / Case Folding

Youtube search... ...Google search

A common strategy is to do case-folding by reducing all letters to lower case. Often this is a good idea: it will allow instances of Automobile at the beginning of a sentence to match with a query of automobile. It will also help on a web search engine when most of your users type in ferrari when they are interested in a Ferrari car. On the other hand, such case folding can equate words that might better be kept apart. Many proper nouns are derived from common nouns and so are distinguished only by case, including companies (General Motors, The Associated Press), government organizations (the Fed vs. fed) and person names (Bush, Black). We already mentioned an example of unintended query expansion with acronyms, which involved not only acronym normalization (C.A.T. $\rightarrow$ CAT) but also case-folding (CAT $\rightarrow$ cat). Capitalization/case-folding | Stanford

Similarity

Youtube search... ...Google search

Word Similarity

Youtube search... ...Google search

Text Clustering

Youtube search... ...Google search

header_short.jpg



Syntax (Parsing)


Syntactic Ambiguity – Sentence is having multiple parse trees.

The set of rules, principles, and processes that govern the structure of sentences (sentence structure) in a given language, usually including word order. The term syntax is also used to refer to the study of such principles and processesFocus on the relationship of the words within a sentence — how a sentence is constructed. In a way, syntax is what we usually refer to as grammar. To derive this understanding, syntactical analysis is usually done at a sentence-level, where as for morphology the analysis is done at word level. When we’re building dependency trees or processing parts-of-speech — we’re basically analyzing the syntax of the sentence.

1*Bi_s86b68I5kDEC2kmU39A.png

Identity Scrubbing

Youtube search... ...Google search

Patient ID: P89474

Mary Phillips is a 45-year-old woman with a history of diabetes. She arrived at New Hope Medical Center on August 5 complaining of abdominal pain. Dr. Gertrude Philippoussis diagnosed her with appendicitis and admitted her at 10 PM.

into this:

Patient ID: [ID]

[NAME] is a [AGE]-year-old woman with a history of diabetes. She arrived at [HOSPITAL] on [DATE] complaining of abdominal pain. Dr. [PHYSICIAN] diagnosed her with appendicitis and admitted her at 10 PM.

or this:

Patient ID: ID586

Sandy Parkinson is a 34-year-old woman with a history of diabetes. She arrived at Mercy Hospital on July 10 complaining of abdominal pain. Dr. Myron Prendergast diagnosed her with appendicitis and admitted her at 10 PM.

Stop Words

Youtube search... ...Google search

One of the major forms of pre-processing is to filter out useless data. In natural language processing, useless words (data), are referred to as stop words. A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query.

Stop-word-removal-using-NLTK.png

Relating Text

Understanding how the words relate to each other and the underlying grammar by segmenting the sentences syntax

Part-of-Speech (POS) Tagging

Youtube search... ...Google search

(POST), also called grammatical tagging or word-category disambiguation, is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech,[1] based on both its definition and its context—i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph. A simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc.

Taggers:

lsmcqqk.jpg

Constituency Tree

Youtube search... ...Google search

a one-to-one-or-more relation; every word in the sentence corresponds to one or more nodes in the tree diagram; employ the convention where the category acronyms (e.g. N, NP, V, VP) are used as the labels on the nodes in the tree. The one-to-one-or-more constituency relation is capable of increasing the amount of sentence structure to the upper limits of what is possible.

E-ICA-01.jpg

Chunking

Youtube search... ...Google search

The Hierarchy of Ideas (also known as chunking) is a linguistic tool used in NLP that allows the speaker to traverse the realms of abstract to specific easily and effortlessly. When we speak or think we use words that indicate how abstract, or how detailed we are in processing the information. In general, as human beings our brain is quite good at chunking information together in order to make it easier for us to process and simpler to understand. Thinking about the word “learning” for example is much simpler that thinking about all the different things that we could be learning about. When we memorise a telephone number or any other sequence of numbers we do not tend to memorise them as separate individual numbers, we group them together to make them easier to remember. Hierarchy of Ideas or Chunking in NLP | Excellence Assured

image-result-for-chunking-examples-in-marketing.gif

Chinking

Youtube search... ...Google search

The process of removing a sequence of tokens from a chunk. If the matching sequence of tokens spans an entire chunk, then the whole chunk is removed; if the sequence of tokens appears in the middle of the chunk, these tokens are removed, leaving two chunks where there was only one before. If the sequence is at the periphery of the chunk, these tokens are removed, and a smaller chunk remains.

chinking.png



Semantics


  • Semantic Ambiguity – Sentence having multiple meanings

The linguistic and philosophical study of meaning in language, programming languages, formal logics, and semiotics. It is concerned with the relationship between signifiers—like words, phrases, signs, and symbols—and what they stand for in reality, their denotation. The “meaning” of the sentence. It’s difficult to (i) link raw text to a knowledge base of mathematical facts in our system and (ii) combine pieces of knowledge together to infer an answer. Meaning in Natural Language is a multi-facetted concept with:

  • semantic,
  • pragmatic,
  • cognitive and
  • social aspects.

Word Embeddings

Youtube search... ...Google search

The collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers.

Example-of-embedding-of-time-series-xi-from-the-temporal-space-left-into-the-pairwise.png

Named Entity Recognition (NER)

Youtube search... ...Google search

Named Entities (also known as entity identification, entity chunking, sequence tagging, Part-of-Speech (POS) Tagging, and entity chunking/extraction) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. Most research on NER systems has been structured as taking an unannotated block of text, and producing an annotated block of text that highlights the names of entities.

maxthonsnap20170219114627.png

Semantic Slot Filling

Youtube search... ...Google search

One way of making sense of a piece of text is to tag the words or tokens which carry meaning to the sentences. There are three main approaches to solve this problem:

  1. Rule Based; regular expressions, context-free grammars
  2. Probabilistic modeling & Machine Learning; likelihood maximization, linear classifiers
  3. Deep Learning; recurrent neural networks, convolutional neural networks

1*bpkI_VpU1j0wXv4VqW5e5Q.png

Relation Extraction

Youtube search... ...Google search

task of extracting semantic relationships from a text. Extracted relationships usually occur between two or more entities of a certain type (e.g. Person, Organisation, Location) and fall into a number of semantic categories (e.g. married to, employed by, lives in). Relationship Extraction

TypicalRelationExtractionPipeline.png

Relation Extraction | Jun Tian



Discourse (Dialog)

Discourse is the creation and organization of the segments of a language above as well as below the sentence. It is segments of language which may be bigger or smaller than a single sentence but the adduced meaning is always beyond the sentence.

  • identifying the discourse structure of a connected text, i.e. the nature of the discourse relationships between sentences (e.g. elaboration, explanation, contrast).
  • recognizing and classifying the speech acts in a chunk of text (e.g. yes-no question, content question, statement, assertion, etc.).

The objects of discourse analysis (discourse, writing, conversation, communicative event) are variously defined in terms of coherent sequences of sentences, propositions, speech, or turns-at-talk.

The problems addressed in discourse research aim to answer two general kinds of questions:

  1. what information is contained in extended sequences of utterances that goes beyond the meaning of the individual utterances themselves?
  2. how does the context in which an utterance is used a�ect the meaning of the individual utterances, or parts of them?

Chapter 6 Discourse and Dialogue



Compared to traditional linguistics Contrary to much of traditional linguistics, discourse analysts not only study language use 'beyond the sentence boundary' but also prefer to analyze 'naturally occurring' language use, not invented examples. Text linguistics is a closely related field. The essential difference between discourse analysis and text linguistics is that discourse analysis aims at revealing socio-psychological characteristics of a person/persons rather than text structure.



Pragmatics

Anaphoric Ambiguity – Phrase or word which is previously mentioned but has a different meaning.

Studies the ways in which context contributes to meaning. Pragmatics encompasses speech act theory, conversational implicature, talk in interaction and other approaches to language behavior in philosophy, sociology, linguistics and anthropology -- Understanding the text as a whole. Popular problems that we’re trying to solve at this stage are:

  • Topic modelling
  • Coreference/Anaphora
  • Summarization
  • Question & Answering

Grouping

Sentence/Document Similarity

Youtube search... ...Google search

Word embeddings have become widespread in Natural Language Processing. They allow us to easily compute the semantic similarity between two words, or to find the words most similar to a target word. However, often we're more interested in the similarity between two sentences or short texts. Comparing Sentence Similarity Methods | Yves Peirsman - NLPtown

image-1.png

Text Classification

Youtube search... ...Google search

Tasks:

  • predict tags or categories
  • predict sentiment
  • filter spam eMails

Text Classification approaches:

  • Flat - there is no inherent hierarchy between the possible categories the data can belong to (or we chose to ignore it). Train either a single classifier to predict all of the available classes or one classifier per category (1 vs All)
  • Hierarchically - organizing the classes, creating a tree or DAG (Directed Acyclic Graph) of categories, exploiting the information on relationships among them. Although there are different types of hierarchical classification approaches, the difference between both modes of reasoning and analysing are particularly easy to understand in these illustrations, taken from a great review on the subject by Silla and Freitas (2011). Taking a top-down approach, training a classifier per level (or node) of the tree (again, although this is not the only hierarchical approach, it is definitely the most widely used and the one we’ve selected for our problem at hands), where a given decision will lead us down a different classification path.

1.png 2.png

Topic Modeling

Youtube search... ...Google search

A type of statistical modeling for discovering the abstract “topics” that occur in a collection of documents. Latent Dirichlet Allocation (LDA) is an example of topic model and is used to classify text in a document to a particular topic

40064_2016_3252_Fig5_HTML.gif


Neural Coreference

Youtube search... ...Google search

Coreference is the fact that two or more expressions in a text – like pronouns or nouns – link to the same person or thing. It is a classical Natural language processing task, that has seen a revival of interest in the past two years as several research groups applied cutting-edge deep-learning and reinforcement-learning techniques to it. It is also one of the key building blocks to building conversational Artificial intelligence.


Whole Word Masking

Youtube search... ...Google search

Training the language model in Bidirectional Encoder Representations from Transformers (BERT) is done by predicting 15% of the tokens in the input, that were randomly picked. These tokens are pre-processed as follows — 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The intuition that led the authors to pick this approach is as follows (Thanks to Jacob Devlin from Google for the insight):

  • If we used [MASK] 100% of the time the model wouldn’t necessarily produce good token representations for non-masked words. The non-masked tokens were still used for context, but the model was optimized for predicting masked words.
  • If we used [MASK] 90% of the time and random words 10% of the time, this would teach the model that the observed word is never correct.
  • If we used [MASK] 90% of the time and kept the same word 10% of the time, then the model could just trivially copy the non-contextual embedding.

Managed Vocabularies

Datasets

Corpora

Youtube search... ...Google search

A corpus (plural corpora) or text corpus is a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules within a specific language territory.

Ontology

Youtube search... ...Google search

(aka knowledge graph) can incorporate computable descriptions that can bring insight in a wide set of compelling applications including more precise knowledge capture, semantic data integration, sophisticated query answering, and powerful association mining - thereby delivering key value for health care and the life sciences. Ontology learning (ontology extraction, ontology generation, or ontology acquisition) is the automatic or semi-automatic creation of ontologies, including extracting the corresponding domain's terms and the relationships between the concepts that these terms represent from a corpus of natural language text, and encoding them with an ontology language for easy retrieval. As building ontologies manually is extremely labor-intensive and time-consuming, there is great motivation to automate the process. Typically, the process starts by extracting terms and concepts or noun phrases from plain text using linguistic processors such as part-of-speech tagging and phrase chunking. Then statistical or symbolic techniques are used to extract relation signatures, often based on pattern-based or definition-based hypernym extraction techniques. Ontology learning | Wikipedia

ontology-dimensions-map_20070423b.png

Taxonomy

Youtube search... ...Google search

The practice and science of classification of things or concepts, including the principles that underlie such classification.

Taxonomy-of-linguistic-analyzers.png

Related

Workbench / Pipeline

Youtube search... ...Google search

technology-stack1.png 4-Figure3-1.png A-global-model-of-the-Power-Workbench.png

Sapir Whorf Hypothesis - How language shapes the way we think

Youtube search... ...Google search

General Natural Language Processing (NLP) Videos