Term Frequency, Inverse Document Frequency (TF-IDF) - Revision history

BPeat at 19:34, 28 April 2023

2023-04-28T19:34:42Z

2023-03-17T12:39:10Z

2023-02-07T01:41:15Z

2019-07-23T21:00:48Z

2019-07-18T16:11:23Z

2019-07-18T16:09:17Z

2019-07-18T16:03:26Z

2019-07-12T18:51:25Z

2019-06-08T16:16:10Z

2019-06-08T16:13:47Z

@@ Line 1: / Line 1: @@
 [http://www.youtube.com/results?search_query=tf+idf+Term+Frequency+Inverse+Document+nlp+nli+natural+language Youtube search...]
-* [[Natural Language Processing (NLP)]]
+* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]]  ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ...  [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
 * [[Python#scikit-learn|scikit-learn]]
 * [[Bag-of-Words (BoW)]]

@@ Line 19: / Line 19: @@
 === Similarity ===
-TF-IDF is a transformation applied to texts to get two real-valued vectors in vector space. We can then obtain the Cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Cosine similarity is a measure of similarity between two non-zero vectors. Using this formula we can find out the similarity between any two documents d1 and d2.  [https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e Building a Simple] [[Assistants|Chatbot]] from Scratch in Python (using NLTK) |  Paul Pandey - Analytics Vidhya
+TF-IDF is a transformation applied to texts to get two real-valued vectors in vector space. We can then obtain the Cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Cosine similarity is a measure of similarity between two non-zero vectors. Using this formula we can find out the similarity between any two documents d1 and d2.  [https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e Building a Simple] [[Assistants#Chatbot | Chatbot]] from Scratch in Python (using NLTK) |  Paul Pandey - Analytics Vidhya
 <youtube>hc3DCn8viWs</youtube>

@@ Line 19: / Line 19: @@
 === Similarity ===
-TF-IDF is a transformation applied to texts to get two real-valued vectors in vector space. We can then obtain the Cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Cosine similarity is a measure of similarity between two non-zero vectors. Using this formula we can find out the similarity between any two documents d1 and d2.  [https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e Building a Simple Chatbot from Scratch in Python (using NLTK) |  Paul Pandey - Analytics Vidhya]
+TF-IDF is a transformation applied to texts to get two real-valued vectors in vector space. We can then obtain the Cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Cosine similarity is a measure of similarity between two non-zero vectors. Using this formula we can find out the similarity between any two documents d1 and d2.  [https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e Building a Simple] [[Assistants|Chatbot]] from Scratch in Python (using NLTK) |  Paul Pandey - Analytics Vidhya
 <youtube>hc3DCn8viWs</youtube>

@@ Line 2: / Line 2: @@
 * [[Natural Language Processing (NLP)]]
-* [[Scikit-learn]] Machine Learning in Python, Simple and efficient tools for data mining and data analysis; Built on NumPy, SciPy, and matplotlib
+* [[Python#scikit-learn|scikit-learn]]
 * [[Bag-of-Words (BoW)]]
 * [[Continuous Bag-of-Words (CBoW)]]

@@ Line 11: / Line 11: @@
 This statistic represents words’ importance in each document. We use a word's frequency as a proxy for its importance: if "football" is mentioned 25 times in a document, it might be more important than if it was only mentioned once. We also use the document frequency (the number of documents containing a given word) as a measure of how common the word is. This minimizes the effect of stop-words such as pronouns, or domain-specific language that does not add much information (for example, a word such as "news" that might be present in most documents).
 <youtube>4vT4fzjkGCQ</youtube>
@@ Line 20: / Line 16: @@
 <youtube>hXNbFNCgPfY</youtube>
 <youtube>bPYJi1E9xeM</youtube>
 <youtube>hc3DCn8viWs</youtube>
 <youtube>927YDZH_MLo</youtube>
 <youtube>HW9W6EBytLg</youtube>
 <youtube>FfLo5OHBwTo</youtube>

@@ Line 4: / Line 4: @@
 * [[Scikit-learn]] Machine Learning in Python, Simple and efficient tools for data mining and data analysis; Built on NumPy, SciPy, and matplotlib
 * [[Bag-of-Words (BoW)]]
-* [[Continuous Bag-of-Words (CBOW)]]
+* [[Continuous Bag-of-Words (CBoW)]]
 * [[Word2Vec]]
 * [[Doc2Vec]]

← Older revision	Revision as of 16:16, 8 June 2019
(No difference)

← Older revision		Revision as of 16:13, 8 June 2019
Line 19:		Line 19:
	<youtube>hXNbFNCgPfY</youtube>		<youtube>hXNbFNCgPfY</youtube>
	<youtube>bPYJi1E9xeM</youtube>		<youtube>bPYJi1E9xeM</youtube>
		+	<youtube>hc3DCn8viWs</youtube>
		+	<youtube>927YDZH_MLo</youtube>
		+	<youtube>HW9W6EBytLg</youtube>
		+	<youtube>FfLo5OHBwTo</youtube>