|
|
| Line 22: |
Line 22: |
| | * [[Embedding]] ... [[Fine-tuning]] ... [[Agents#AI-Powered Search|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]] ... [[...find outliers]] | | * [[Embedding]] ... [[Fine-tuning]] ... [[Agents#AI-Powered Search|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]] ... [[...find outliers]] |
| | * [[Prompting vs AI Model Fine-Tuning vs AI Embeddings]] | | * [[Prompting vs AI Model Fine-Tuning vs AI Embeddings]] |
| − | * [[AI Solver]] ... [[Algorithms]] ... [[Algorithm Administration|Administration]] ... [[Model Search]] ... [[Discriminative vs. Generative]] ... [[Optimizer]] ... [[Train, Validate, and Test]]
| |
| − | * [[Math for Intelligence]] ... [[Finding Paul Revere]] ... [[Social Network Analysis (SNA)]] ... [[Dot Product]] ... [[Kernel Trick]]
| |
| − | * [[Capabilities]]:
| |
| − | *** [[AI-Powered Search|Search]] (where results are ranked by relevance to a query string)
| |
| − | *** [[Clustering]] (where text strings are grouped by similarity)
| |
| − | *** [[Recommendation]]s (where items with related text strings are recommended)
| |
| − | *** [[Anomaly Detection]] (where outliers with little relatedness are identified)
| |
| − | *** [[Classification]] (where text strings are classified by their most similar label)
| |
| − | *** [[Dimensional Reduction]]
| |
| − | *** [[...find outliers]] ... diversity measurement (where similarity distributions are analyzed)
| |
| − | * [https://medium.com/@ryanntk/choosing-the-right-embedding-model-a-guide-for-llm-applications-7a60180d28e3 Choosing the Right Embedding Model: A Guide for LLM Applications | Ryan Nguyen - Medium]
| |
| − | * [https://partee.io/2022/08/11/vector-embeddings/ Vector Embeddings: From the Basics to Production | Sam Partee - Partee.IO]
| |
| − | Types:
| |
| − | * [[Local Linear Embedding (LLE)]]
| |
| − | * [[T-Distributed Stochastic Neighbor Embedding (t-SNE)]]
| |
| | | | |
| | | | |
| − | = <span id="AI Encoding & AI Embedding"></span>AI Encoding & AI Embedding =
| |
| − | * [[Excel]] ... [[LangChain#Documents|Documents]] ... [[Database]] ... [[Graph]] ... [[LlamaIndex]]
| |
| − |
| |
| − | The terms "AI encodings" and "AI embeddings" are sometimes used interchangeably, but there is a subtle difference between the two.
| |
| − |
| |
| − | * </b>Encodings</b> are a general term for any representation of data that is used by a [[Machine Learning (ML)]] model. This could be a one-hot encoding, a bag-of-words representation, or a more complex representation such as a word embedding.
| |
| − | * </b>Embeddings</b> are a specific type of AI encoding that is learned from data. Embeddings are typically represented as [[Math for Intelligence#Vector|vectors]] of real numbers, and they capture the meaning and context of the data they represent.
| |
| − |
| |
| − |
| |
| − | In other words, all embeddings are encodings, but not all encodings are embeddings. Here are some examples of AI encodings that are not embeddings:
| |
| − |
| |
| − | * One-hot Encoding is a simple way to represent categorical data as a [[Math for Intelligence#Vector|vector]]. For example, the word "dog" would be represented as a [[Math for Intelligence#Vector|vector]] of 100 zeros, with a single 1 at the index corresponding to the word "dog" in a vocabulary of 100 words.
| |
| − | * Bag-of-words is a more sophisticated way to represent text data as a [[Math for Intelligence#Vector|vector]]. This involves counting the number of times each word appears in a document, and then representing the document as a [[Math for Intelligence#Vector|vector]] of these counts.
| |
| − |
| |
| − |
| |
| − | AI Embeddings are a type of representation of text that captures the meaning of the text. This can be used for tasks such as search, classification, and recommendation. allow the model to search in a “database” and return the best result. Here are some examples of AI Embeddings:
| |
| − |
| |
| − | * Word embeddings are a type of embedding that represents words as [[Math for Intelligence#Vector|vectors]] of real numbers. These [[Math for Intelligence#Vector|vectors]] are typically learned from a large corpus of text, and they capture the meaning and context of the words they represent.
| |
| − | * Image embeddings are a type of embedding that represents images as [[Math for Intelligence#Vector|vectors]] of real numbers. These [[Math for Intelligence#Vector|vectors]] are typically learned from a large dataset of images, and they capture the visual features of the images they represent.
| |
| − |
| |
| − | Embedding...
| |
| − | * projecting an input into another more convenient representation space. For example we can project (embed) faces into a space in which face matching can be more reliable. | [https://www.quora.com/profile/Chomba-Bupe Chomba Bupe]
| |
| − | * a mapping of a discrete — categorical — variable to a [[Math for Intelligence#Vector|vector]] of continuous numbers. In the [[context]] of neural networks, embeddings are low-dimensional, learned continuous [[Math for Intelligence#Vector|vector]] representations of discrete variables. [[Neural Network]] embeddings are useful because they can reduce the dimensionality of categorical variables and meaningfully represent categories in the transformed space. [https://towardsdatascience.com/neural-network-embeddings-explained-4d028e6f0526 Neural Network Embeddings Explained | Will Koehrsen - Towards Data Science]
| |
| − | * a relatively low-dimensional space into which you can translate high-dimensional [[Math for Intelligence#Vector|vectors]]. Embeddings make it easier to do [[Machine Learning (ML)]] on large inputs like sparse [[Math for Intelligence#Vector|vectors]] representing words. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space. An embedding can be learned and reused across models. [https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture Embeddings | Machine Learning Crash Course]
| |
| − | * Search: Embeddings can be used to rank search results by relevance to a query string.
| |
| − | * Clustering: Embeddings can be used to group text strings by similarity.
| |
| − | * Recommendations: Embeddings can be used to recommend items that are related to a user's interests.
| |
| − | * Anomaly detection: Embeddings can be used to identify outliers with little relatedness.
| |
| − | * Diversity measurement: Embeddings can be used to analyze similarity distributions.
| |
| − | * Classification: Embeddings can be used to classify text strings by their most similar label.
| |
| − |
| |
| − |
| |
| − | <hr><center><b><i>
| |
| − |
| |
| − | By employing techniques like Word Embeddings, Sentence Embeddings, or Contextual embedding, [[Math for Intelligence#Vector|vector]] embeddings provide a compact and meaningful representation of textual data. Word embeddings, for instance, map words to fixed-length [[Math for Intelligence#Vector|vectors]], where words with similar meanings are positioned closer to one another in the [[Math for Intelligence#Vector|vector]] space. This allows for efficient semantic search, information retrieval, and language understanding tasks.
| |
| − |
| |
| − | </i></b></center><hr>
| |
| − |
| |
| − |
| |
| − | Embeddings have 3 primary purposes:
| |
| − | # Finding nearest neighbors in the embedding space. These can be used to make recommendations based on user interests or [[Clustering|cluster]] categories.
| |
| − | # As input to a [[Machine Learning (ML)]] model for a supervised task.
| |
| − | # For [[visualization]] of concepts and relations between categories.
| |
| − |
| |
| − | [[Fine-tuning]]
| |
| − | = <span id="Fine-tuning"></span>Fine-tuning =
| |
| | A process of retraining a language model on a new dataset of data. This can be used to improve the model's performance on a specific task, such as generating text, translating languages, or answering questions. Fine-tuning is a way to add new knowledge to an existing AI model. It’s a simple upgrade that allows the model to learn new information. | | A process of retraining a language model on a new dataset of data. This can be used to improve the model's performance on a specific task, such as generating text, translating languages, or answering questions. Fine-tuning is a way to add new knowledge to an existing AI model. It’s a simple upgrade that allows the model to learn new information. |
| | | | |
| Line 106: |
Line 45: |
| | <youtube>y9PHWGOa8HA</youtube> | | <youtube>y9PHWGOa8HA</youtube> |
| | <youtube>HMbctCYJLbw</youtube> | | <youtube>HMbctCYJLbw</youtube> |
| − |
| |
| − | [[Prompting vs AI Model Fine-Tuning vs AI Embeddings]]
| |
| − | = <span id="Prompting vs AI Model Fine-Tuning vs AI Embeddings"></span>Prompting vs AI Model Fine-Tuning vs AI Embeddings =
| |
| − | * [https://coderevolution.ro/knowledge-base/faq/what-is-the-difference-between-ai-model-fine-tuning-and-ai-embeddings What is the difference between AI model fine-tuning and AI Embeddings]
| |
| − | * [https://github.com/openai/openai-cookbook GitHub - openai/openai-cookbook: Examples and guides]
| |
| − | * [https://huggingface.co/blog/how-to-train-sentence-transformers Train and Fine-Tune Sentence Transformers Models | Hugging Face]
| |
| − | * [https://stackoverflow.com/questions/40345607/how-does-fine-tuning-word-embeddings-work How does Fine-tuning Word Embeddings work? | Stack Overflow]
| |
| − | * [https://medium.com/intuitionmachine/ai-development-tradeoffs-using-prompting-fine-tuning-and-search-engine-embeddings-91ff75beb7e2 AI Development Tradeoffs using Prompting, Fine-Tuning, and Search Engine Embeddings | Carlos E. Perez - Medium]
| |
| − |
| |
| − | Beyond simple prompt engineering, there are two design approaches to consider: building an embedding [[database]] of all proprietary content and dynamically searching for relevant information at runtime, or sending the content to the AI provider to fine-tune the model.
| |
| − |
| |
| − | <table>
| |
| − | <tr>
| |
| − | <th>Feature</th>
| |
| − | <th>AI Model Fine-tuning</th>
| |
| − | <th>AI Embeddings</th>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Purpose</td>
| |
| − | <td>Improve the performance of a language model on a specific task</td>
| |
| − | <td>Capture the meaning of text</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Process</td>
| |
| − | <td>Retrain the language model on a new dataset of data</td>
| |
| − | <td>Calculate a numerical representation of the text</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Applications</td>
| |
| − | <td>Text generation, translation, question answering</td>
| |
| − | <td>Search, classification, recommendation</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Advantages</td>
| |
| − | <td>Can improve the performance of a language model significantly</td>
| |
| − | <td>Efficient and easy to use</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Disadvantages</td>
| |
| − | <td>Can be time-consuming and expensive</td>
| |
| − | <td>May not be as accurate as fine-tuning</td>
| |
| − | </tr>
| |
| − | </table>
| |
| − |
| |
| − |
| |
| − | <hr><center><b><i>
| |
| − |
| |
| − | The best technique to use will depend on the specific task you want to perform and the resources you have available
| |
| − |
| |
| − | </i></b></center><hr>
| |
| − |
| |
| − |
| |
| − |
| |
| − | * <b>Prompting</b>: is the simplest technique. It involves providing the LLM with a text prompt that describes the task you want it to perform. The LLM then generates text that is consistent with the prompt. This is a very efficient technique, as it does not require any retraining of the LLM. However, it can be less accurate than fine-tuning or embeddings, as the LLM may not be able to understand the prompt perfectly.
| |
| − |
| |
| − | * <b>Fine-tuning</b>: is a more powerful technique than prompting. It involves retraining the LLM on a dataset of examples for the specific task you want it to perform. This can improve the accuracy of the LLM, but it also requires more training data and compute resources.
| |
| − |
| |
| − | * <b>Embeddings</b>: are a middle ground between prompting and fine-tuning. They involve using a small model to learn a representation of the input text. This representation is then used to initialize the LLM, which can then be fine-tuned on a dataset of examples for the specific task you want it to perform. This can improve the accuracy of the LLM over prompting, without requiring as much training data or compute resources as fine-tuning.
| |
| − |
| |
| − |
| |
| − | <table>
| |
| − | <tr>
| |
| − | <th>Technique</th>
| |
| − | <th>Performance</th>
| |
| − | <th>Efficiency</th>
| |
| − | <th>Flexibility</th>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Prompting</td>
| |
| − | <td>Low</td>
| |
| − | <td>High</td>
| |
| − | <td>High</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Fine-tuning</td>
| |
| − | <td>High</td>
| |
| − | <td>Low</td>
| |
| − | <td>Low</td>
| |
| − | </tr>
| |
| − | <tr>
| |
| − | <td>Embeddings</td>
| |
| − | <td>Medium</td>
| |
| − | <td>Medium</td>
| |
| − | <td>Medium</td>
| |
| − | </tr>
| |
| − | </table>
| |
| − |
| |
| − |
| |
| − | <youtube>7rwiewDGHsk</youtube>
| |
| − | <youtube>_wtfszFzqt0</youtube>
| |
| − |
| |
| − | = OpenAI Note =
| |
| − | * [https://openai.com/blog/new-and-improved-embedding-model New and improved embedding model] ... We are excited to announce a new embedding model which is significantly more capable, cost effective, and simpler to use.
| |
| − | * [https://platform.openai.com/docs/guides/embeddings Embeddings]
| |
| − |
| |
| − | Embeddings are a numerical representation of text that can be used to measure the relateness between two pieces of text. Our second generation embedding model, text-embedding-ada-002 is a designed to replace the previous 16 first-generation embedding models at a fraction of the cost. An embedding is a [[Math for Intelligence#Vector|vector]] (list) of floating point numbers. The distance between two [[Math for Intelligence#Vector|vectors]] measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness.
| |
| − |
| |
| − | <youtube>ySus5ZS0b94</youtube>
| |
