Semantic Search - Revision history

BPeat: Text replacement - "* Conversational AI ... ChatGPT | OpenAI ... Bing | Microsoft ... Bard | Google ... Claude | Anthropic ... Perplexity ... You ... Ernie | Baidu" to "* Conversational AI ... [[C...

2024-03-16T15:39:02Z

BPeat at 17:17, 9 October 2023

2023-10-09T17:17:39Z

BPeat at 17:13, 9 October 2023

2023-10-09T17:13:04Z

BPeat at 17:12, 9 October 2023

2023-10-09T17:12:19Z

BPeat at 17:09, 9 October 2023

2023-10-09T17:09:06Z

BPeat: /* Semantic Search vs Lexical Search */

2023-10-09T17:08:22Z

‎Semantic Search vs Lexical Search

BPeat at 17:02, 9 October 2023

2023-10-09T17:02:50Z

BPeat: /* Embeddings and Similarity */

2023-10-09T17:01:58Z

‎Embeddings and Similarity

BPeat: /* Limitations */

2023-10-09T16:56:54Z

‎Limitations

BPeat at 16:51, 9 October 2023

2023-10-09T16:51:18Z

@@ Line 23: / Line 23: @@
 * [[Embedding]] ... [[Fine-tuning]] ... [[Retrieval-Augmented Generation (RAG)|RAG]] ... [[Agents#AI-Powered Search|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]].  [[...find outliers]]
 * [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]]  ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ...  [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
-* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing]] | [[Microsoft]] ... [[Bard]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[Ernie]] | [[Baidu]]
+* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Ernie]] | [[Baidu]]
 * [[Papers Search]]
 * [[Google Semantic Reactor]]

@@ Line 32: / Line 32: @@
 * <b>[[Natural Language Processing (NLP)]]</b>: [[Natural Language Processing (NLP)|NLP techniques]] can be used to extract the meaning from the search query and the documents being searched.
 * <b>[[Embedding|Text embeddings]]</b>: [[Embedding|Text embeddings]] are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Text embeddings are an essential part of semantic search. They allow semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. This is because [[Embedding|text embeddings]] are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
-* <b>Similarity metrics</B>: are used to compare the meaning of different pieces of text.
+* <b>Similarity metrics</B>: are used to compare the meaning of different pieces of text. A similarity metric is a measure of how similar two pieces of text are. There are many different similarity metrics, but some of the most common ones include cosine similarity, Euclidean distance, and Jaccard similarity.
 = Similarity =
-Similarity metrics are used to compare the meaning of different pieces of text. Which similarity metric is best to use depends on the specific task at hand. For example, cosine similarity is often used to compare the meaning of text embeddings, while Euclidean distance is often used to compare the features of different pieces of text. There are many different similarity metrics, but some of the most common ones include:
+Which similarity metric is best to use depends on the specific task at hand. For example, cosine similarity is often used to compare the meaning of text [[embedding]]s, while Euclidean distance is often used to compare the features of different pieces of text. There are many different similarity metrics, but some of the most common ones include:
 * <b>Cosine similarity</b>: Cosine similarity is a measure of the similarity between two vectors. It is calculated by taking the dot product of the two vectors and dividing by the product of their magnitudes. Cosine similarity is often used to compare the meaning of text embeddings.

@@ Line 34: / Line 34: @@
 * <b>Similarity metrics</B>: are used to compare the meaning of different pieces of text.
-== Similarity ==
 Similarity metrics are used to compare the meaning of different pieces of text. Which similarity metric is best to use depends on the specific task at hand. For example, cosine similarity is often used to compare the meaning of text embeddings, while Euclidean distance is often used to compare the features of different pieces of text. There are many different similarity metrics, but some of the most common ones include:
@@ Line 42: / Line 43: @@
-== Embeddings & Similarity ==
+= Embeddings & Similarity =
 [[Embedding]]s and similarity are two different concepts, but they are both important for semantic search, as they allow the search engine to understand the meaning of the search query,  the content of the documents being searched, and to return the most relevant results. [[Embedding]]s are a way of representing text in a numerical format, and similarity is a measure of how similar two pieces of text are.

@@ Line 32: / Line 32: @@
 * <b>[[Natural Language Processing (NLP)]]</b>: [[Natural Language Processing (NLP)|NLP techniques]] can be used to extract the meaning from the search query and the documents being searched.
 * <b>[[Embedding|Text embeddings]]</b>: [[Embedding|Text embeddings]] are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Text embeddings are an essential part of semantic search. They allow semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. This is because [[Embedding|text embeddings]] are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
+* <b>Similarity metrics</B>: are used to compare the meaning of different pieces of text.
 == Similarity ==
@@ Line 67: / Line 59: @@
 In this example, the [[embedding]]s allow the search engine to understand that the user is looking for information on how to make a cake, even though the search query does not contain the word "recipe". The similarity metric allows the search engine to identify the documents that are most relevant to the user's query, even if they use different words or have different sentence structures.
 = Limitations =
@@ Line 89: / Line 82: @@
 * A semantic search engine that uses a similarity metric that is inaccurate for complex or ambiguous text may return irrelevant results for those types of queries.
 * A semantic search engine that uses a similarity metric that cannot capture all aspects of semantic similarity may miss some relevant results.

@@ Line 30: / Line 30: @@
 <u>Semantic search</u> is a type of search that tries to understand the meaning of the search query and the content of the documents being searched, in order to return the most relevant results. Semantic search uses a variety of techniques, including:
-* <b>[[Natural Language Processing (NLP)]]</b>: NLP techniques can be used to extract the meaning from the search query and the documents being searched.
+* <b>[[Natural Language Processing (NLP)]]</b>: [[Natural Language Processing (NLP)|NLP techniques]] can be used to extract the meaning from the search query and the documents being searched.
 * <b>[[Embedding|Text embeddings]]</b>: [[Embedding|Text embeddings]] are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Text embeddings are an essential part of semantic search. They allow semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. This is because [[Embedding|text embeddings]] are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.

@@ Line 41: / Line 41: @@
 * <b>With semantic search</b>, the search engine would use NLP techniques to understand that you are looking for information about how to bake a cake. It would then use [[Embedding|text embeddings]] to compare the meaning of the search query to the meaning of the documents in its index. This would allow the search engine to return the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake. For example, the [[Embedding|text embeddings]] for the words "cake" and "dessert" would be very similar, because these words are semantically related. This means that a semantic search algorithm would be able to identify documents that are relevant to the search query "cake", even if they do not contain the keyword "dessert".
 == Embeddings & Similarity ==
@@ Line 53: / Line 61: @@
 Here is an example of how [[embedding]]s and similarity are used in semantic search:
-* The user enters the search query "how to bake a cake".
+* The user enters the search query <b><i>"how to bake a cake"</i></b>.
 * The search engine uses [[embedding]]s to represent the meaning of the search query and the content of the documents in its index.
 * The search engine then uses a similarity metric to compare the meaning of the search query to the meaning of the documents in the index.

@@ Line 40: / Line 40: @@
 * <b>With semantic search</b>, the search engine would use NLP techniques to understand that you are looking for information about how to bake a cake. It would then use [[Embedding|text embeddings]] to compare the meaning of the search query to the meaning of the documents in its index. This would allow the search engine to return the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake. For example, the [[Embedding|text embeddings]] for the words "cake" and "dessert" would be very similar, because these words are semantically related. This means that a semantic search algorithm would be able to identify documents that are relevant to the search query "cake", even if they do not contain the keyword "dessert".
-== Embeddings and Similarity ==
+== Embeddings & Similarity ==
-Embeddings and similarity are two different concepts, but they are both important for semantic search. Embeddings are a way of representing text in a numerical format, and similarity is a measure of how similar two pieces of text are. Embeddings and similarity are both important for semantic search, as they allow the search engine to understand the meaning of the search query and the content of the documents being searched, and to return the most relevant results.
+[[Embedding]]s and similarity are two different concepts, but they are both important for semantic search, as they allow the search engine to understand the meaning of the search query,  the content of the documents being searched, and to return the most relevant results. [[Embedding]]s are a way of representing text in a numerical format, and similarity is a measure of how similar two pieces of text are.
-Embeddings are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Embeddings are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
+* [[Embedding]]s are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. [[Embedding]]s are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
-Similarity is a measure of how similar two pieces of text are. Similarity metrics can be used to compare the meaning of different pieces of text, or they can be used to compare the features of different pieces of text, such as their word order or sentence structure.
+* Similarity is a measure of how similar two pieces of text are. Similarity metrics can be used to compare the meaning of different pieces of text, or they can be used to compare the features of different pieces of text, such as their word order or sentence structure.
-In the context of semantic search, embeddings are used to represent the meaning of the search query and the content of the documents being searched. Similarity metrics are then used to compare the meaning of the search query to the meaning of the documents in the index. This allows the search engine to return the most relevant results, even if they do not contain the exact same words as the search query.
+In the context of semantic search, [[embedding]]s are used to represent the meaning of the search query and the content of the documents being searched. Similarity metrics are then used to compare the meaning of the search query to the meaning of the documents in the index. This allows the search engine to return the most relevant results, even if they do not contain the exact same words as the search query.
-Here is an example of how embeddings and similarity are used in semantic search:
+Here is an example of how [[embedding]]s and similarity are used in semantic search:
 * The user enters the search query "how to bake a cake".
-* The search engine uses embeddings to represent the meaning of the search query and the content of the documents in its index.
+* The search engine uses [[embedding]]s to represent the meaning of the search query and the content of the documents in its index.
 * The search engine then uses a similarity metric to compare the meaning of the search query to the meaning of the documents in the index.
 * The search engine returns the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake.
-In this example, the embeddings allow the search engine to understand that the user is looking for information on how to make a cake, even though the search query does not contain the word "recipe". The similarity metric allows the search engine to identify the documents that are most relevant to the user's query, even if they use different words or have different sentence structures.
+In this example, the [[embedding]]s allow the search engine to understand that the user is looking for information on how to make a cake, even though the search query does not contain the word "recipe". The similarity metric allows the search engine to identify the documents that are most relevant to the user's query, even if they use different words or have different sentence structures.
 = Limitations =

@@ Line 39: / Line 39: @@
 * <b>With semantic search</b>, the search engine would use NLP techniques to understand that you are looking for information about how to bake a cake. It would then use [[Embedding|text embeddings]] to compare the meaning of the search query to the meaning of the documents in its index. This would allow the search engine to return the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake. For example, the [[Embedding|text embeddings]] for the words "cake" and "dessert" would be very similar, because these words are semantically related. This means that a semantic search algorithm would be able to identify documents that are relevant to the search query "cake", even if they do not contain the keyword "dessert".
 = Limitations =
 [[Embedding]]s and similarity are powerful tools for semantic search, but they have some limitations.
-== Limitations of embeddings ==
+== Limitations of Embeddings ==
 * <b>[[Embedding]]s can be biased</b>. [[Embedding]]s are trained on a corpus of text, which may reflect the biases of the authors of that text. This means that [[embedding]]s may learn to represent certain words or concepts in a more positive or negative light than others.
@@ Line 49: / Line 67: @@
 * <b>[[Embedding]]s can be computationally expensive</b>. Training and using [[embedding]]s can be computationally expensive, especially for large datasets.
-== Limitations of similarity ==
+== Limitations of Similarity ==
 * <b>Similarity metrics can be inaccurate</b>. Similarity metrics are used to compare the meaning of different pieces of text. However, these metrics can be inaccurate, especially for text that is complex or ambiguous.

@@ Line 31: / Line 31: @@
 * <b>[[Natural Language Processing (NLP)]]</b>: NLP techniques can be used to extract the meaning from the search query and the documents being searched.
-* <b>[[Embedding|Text embeddings]]</b>: Text embeddings are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Text embeddings are an essential part of semantic search. They allow semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. This is because text embeddings are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
+* <b>[[Embedding|Text embeddings]]</b>: [[Embedding|Text embeddings]] are a way of representing text in a numerical format. This allows semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. Text embeddings are an essential part of semantic search. They allow semantic search algorithms to compare the meaning of different pieces of text, even if they use different words. This is because [[Embedding|text embeddings]] are trained on a large corpus of text, and they learn to represent similar pieces of text in a similar way.
 = Semantic Search vs Lexical Search =
 One way to think about the difference between <u>semantic search</u> and <u>lexical search</u> is to imagine that you are looking for information about <b><i>how to make a cake.</i></b>
-* <b>With lexical search</b>, you would enter the keywords <b><i>"make cake"</i></b>
+* <b>With lexical search</b>, you would enter the <u>keywords</u> <b><i>"make cake"</i></b> into the search engine. The search engine would then return all of the documents that contain those keywords. This might include documents about making different types of cakes, as well as documents about other topics, such as cake decorating or cake recipes. Lexical search, which simply matches keywords in the query to keywords in the documents.
-* <b>With semantic search</b>, the search engine would use NLP techniques to understand that you are looking for information about how to bake a cake. It would then use text embeddings to compare the meaning of the search query to the meaning of the documents in its index. This would allow the search engine to return the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake. For example, the text embeddings for the words "cake" and "dessert" would be very similar, because these words are semantically related. This means that a semantic search algorithm would be able to identify documents that are relevant to the search query "cake", even if they do not contain the keyword "dessert".
+* <b>With semantic search</b>, the search engine would use NLP techniques to understand that you are looking for information about how to bake a cake. It would then use [[Embedding|text embeddings]] to compare the meaning of the search query to the meaning of the documents in its index. This would allow the search engine to return the most relevant documents, such as recipes for different types of cakes or instructions on how to bake a cake. For example, the [[Embedding|text embeddings]] for the words "cake" and "dessert" would be very similar, because these words are semantically related. This means that a semantic search algorithm would be able to identify documents that are relevant to the search query "cake", even if they do not contain the keyword "dessert".
 = Limitations =
-Embeddings and similarity are powerful tools for semantic search, but they have some limitations.
+[[Embedding]]s and similarity are powerful tools for semantic search, but they have some limitations.
 == Limitations of embeddings ==
-* <b>Embeddings can be biased</b>. Embeddings are trained on a corpus of text, which may reflect the biases of the authors of that text. This means that embeddings may learn to represent certain words or concepts in a more positive or negative light than others.
+* <b>[[Embedding]]s can be biased</b>. [[Embedding]]s are trained on a corpus of text, which may reflect the biases of the authors of that text. This means that [[embedding]]s may learn to represent certain words or concepts in a more positive or negative light than others.
-* <b>Embeddings cannot capture all aspects of meaning</b>. Embeddings are a numerical representation of text, and they cannot capture all of the nuances of human language. For example, embeddings may not be able to capture the difference between the different meanings of a word, such as the word "bank" (as in a financial institution) or "bank" (as in the side of a river).
+* <b>[[Embedding]]s cannot capture all aspects of meaning</b>. [[Embedding]]s are a numerical representation of text, and they cannot capture all of the nuances of human language. For example, [[embedding]]s may not be able to capture the difference between the different meanings of a word, such as the word "bank" (as in a financial institution) or "bank" (as in the side of a river).
-* <b>Embeddings can be computationally expensive</b>. Training and using embeddings can be computationally expensive, especially for large datasets.
+* <b>[[Embedding]]s can be computationally expensive</b>. Training and using [[embedding]]s can be computationally expensive, especially for large datasets.
 == Limitations of similarity ==
@@ Line 55: / Line 54: @@
 * <b>Similarity metrics may not be able to capture all aspects of semantic similarity</b>. Semantic similarity is a complex concept, and similarity metrics may not be able to capture all of its aspects. For example, two pieces of text may be semantically similar even if they do not use the same words or have the same structure.
-Despite these limitations, embeddings and similarity are still powerful tools for semantic search. By using these techniques, semantic search algorithms can achieve better results than traditional lexical search algorithms.
+Despite these limitations, [[embedding]]s and similarity are still powerful tools for semantic search. By using these techniques, semantic search algorithms can achieve better results than traditional lexical search algorithms.
-Here are some examples of how the limitations of embeddings and similarity can impact semantic search:
+Here are some examples of how the limitations of [[embedding]]s and similarity can impact semantic search:
-* A semantic search engine that uses embeddings that are biased against certain groups of people may return less relevant results for those groups.
+* A semantic search engine that uses [[embedding]]s that are biased against certain groups of people may return less relevant results for those groups.
 * A semantic search engine that uses a similarity metric that is inaccurate for complex or ambiguous text may return irrelevant results for those types of queries.
 * A semantic search engine that uses a similarity metric that cannot capture all aspects of semantic similarity may miss some relevant results.