Retrieval-Augmented Generation (RAG) - Revision history

BPeat at 21:08, 28 April 2024

2024-04-28T21:08:27Z

BPeat at 22:01, 19 April 2024

2024-04-19T22:01:02Z

BPeat at 21:56, 19 April 2024

2024-04-19T21:56:02Z

BPeat at 21:51, 19 April 2024

2024-04-19T21:51:52Z

BPeat at 21:49, 19 April 2024

2024-04-19T21:49:03Z

BPeat at 11:51, 13 October 2023

2023-10-13T11:51:18Z

BPeat at 10:10, 10 October 2023

2023-10-10T10:10:40Z

BPeat: /* RAG Application Evaluation */

2023-09-26T10:57:31Z

‎RAG Application Evaluation

BPeat: /* RAG Application Evaluation */

2023-09-26T10:39:12Z

‎RAG Application Evaluation

BPeat at 10:34, 26 September 2023

2023-09-26T10:34:10Z

@@ Line 120: / Line 120: @@
 * Use a variety of metrics to assess the performance of the application. This will help you to get a more complete picture of the application's strengths and weaknesses.
 * Use a held-out test set to evaluate the application on data that it has not seen before. This will help you to get a more accurate estimate of the application's performance in the real world.
-* Consider using a user-study to evaluate the application from the user's perspective. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.
+* Consider using a user-study to evaluate the application from the user's [[perspective]]. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.
 <youtube>mE7IDf2SmJg</youtube>

@@ Line 122: / Line 122: @@
 * Consider using a user-study to evaluate the application from the user's perspective. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.
 <youtube>u47GtXwePms</youtube>
-<youtube>mE7IDf2SmJg</youtube>
+<youtube>P8tOjiYEFqU</youtube>
 = RAG from Scratch =

@@ Line 121: / Line 121: @@
 * Use a held-out test set to evaluate the application on data that it has not seen before. This will help you to get a more accurate estimate of the application's performance in the real world.
 * Consider using a user-study to evaluate the application from the user's perspective. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.
 = RAG from Scratch =

@@ Line 26: / Line 26: @@
 * [https://howaibuildthis.substack.com/p/rag-pipeline-pitfalls-the-untold RAG Pipeline Pitfalls: The Untold Challenges of Embedding Table | Ryan Nguyen - How AI Built This] ... A typical journey of building the RAG pipeline from Zero to Something and a guide on handling tables of RAG with LlamaIndex
 * [https://towardsdatascience.com/forget-rag-the-future-is-rag-fusion-1147298d8ad1 Forget RAG, the Future is RAG-Fusion | Adrian H. Raudaschl - Towards Data Science] ... The Next Frontier of Search: Retrieval Augmented Generation meets Reciprocal Rank Fusion and Generated Queries
-* [https://www.freecodecamp.org/news/mastering-rag-from-scratch/ Learn RAG from Scratch – Python AI Tutorial from a LangChain Engineer | Beau Carnes - freeCodeCamp]
@@ Line 121: / Line 121: @@
 * Use a held-out test set to evaluate the application on data that it has not seen before. This will help you to get a more accurate estimate of the application's performance in the real world.
 * Consider using a user-study to evaluate the application from the user's perspective. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.

← Older revision		Revision as of 21:49, 19 April 2024
Line 26:		Line 26:
	* [https://howaibuildthis.substack.com/p/rag-pipeline-pitfalls-the-untold RAG Pipeline Pitfalls: The Untold Challenges of Embedding Table \| Ryan Nguyen - How AI Built This] ... A typical journey of building the RAG pipeline from Zero to Something and a guide on handling tables of RAG with LlamaIndex		* [https://howaibuildthis.substack.com/p/rag-pipeline-pitfalls-the-untold RAG Pipeline Pitfalls: The Untold Challenges of Embedding Table \| Ryan Nguyen - How AI Built This] ... A typical journey of building the RAG pipeline from Zero to Something and a guide on handling tables of RAG with LlamaIndex
	* [https://towardsdatascience.com/forget-rag-the-future-is-rag-fusion-1147298d8ad1 Forget RAG, the Future is RAG-Fusion \| Adrian H. Raudaschl - Towards Data Science] ... The Next Frontier of Search: Retrieval Augmented Generation meets Reciprocal Rank Fusion and Generated Queries		* [https://towardsdatascience.com/forget-rag-the-future-is-rag-fusion-1147298d8ad1 Forget RAG, the Future is RAG-Fusion \| Adrian H. Raudaschl - Towards Data Science] ... The Next Frontier of Search: Retrieval Augmented Generation meets Reciprocal Rank Fusion and Generated Queries
		+	* [https://www.freecodecamp.org/news/mastering-rag-from-scratch/ Learn RAG from Scratch – Python AI Tutorial from a LangChain Engineer \| Beau Carnes - freeCodeCamp]

@@ Line 25: / Line 25: @@
 * [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag LlamaIndex: How to evaluate your RAG applications | Ryan Nguyen - How AI Built This]
 * [https://howaibuildthis.substack.com/p/rag-pipeline-pitfalls-the-untold RAG Pipeline Pitfalls: The Untold Challenges of Embedding Table | Ryan Nguyen - How AI Built This] ... A typical journey of building the RAG pipeline from Zero to Something and a guide on handling tables of RAG with LlamaIndex
@@ Line 42: / Line 43: @@
 * It can be used to generate text on a variety of topics, even if the model has not been trained on that specific topic.
 * It is a more efficient and transparent approach than fine-tuning, as it does not require a large amount of labeled data.
 However, RAG also has some limitations:

← Older revision		Revision as of 10:10, 10 October 2023
Line 21:		Line 21:

	* [[Embedding]] ... [[Fine-tuning]] ... [[Retrieval-Augmented Generation (RAG)\|RAG]] ... [[Agents#AI-Powered Search\|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]]. [[...find outliers]]		* [[Embedding]] ... [[Fine-tuning]] ... [[Retrieval-Augmented Generation (RAG)\|RAG]] ... [[Agents#AI-Powered Search\|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]]. [[...find outliers]]
		+	* [[Excel]] ... [[LangChain#Documents\|Documents]] ... [[Database\|Database; Vector & Relational]] ... [[Graph]] ... [[LlamaIndex]]
	* [https://howaibuildthis.substack.com/p/so-you-want-to-improve-your-rag-pipeline So you want to improve your RAG pipeline \| Ryan Nguyen - How AI Built This] ... Ways to go from prototype to production with LlamaIndex		* [https://howaibuildthis.substack.com/p/so-you-want-to-improve-your-rag-pipeline So you want to improve your RAG pipeline \| Ryan Nguyen - How AI Built This] ... Ways to go from prototype to production with LlamaIndex
		+	* [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag LlamaIndex: How to evaluate your RAG applications \| Ryan Nguyen - How AI Built This]
		+	* [https://howaibuildthis.substack.com/p/rag-pipeline-pitfalls-the-untold RAG Pipeline Pitfalls: The Untold Challenges of Embedding Table \| Ryan Nguyen - How AI Built This] ... A typical journey of building the RAG pipeline from Zero to Something and a guide on handling tables of RAG with LlamaIndex

@@ Line 90: / Line 90: @@
 = RAG Application Evaluation =
 * [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag?utm_source=post-email-title&publication_id=1643556&post_id=137378927 LlamaIndex: How to evaluate your RAG applications | Ryan Nguyen - How AI Built This] ... There are a few tools in the market that can help you to have a deep evaluation of your app such as [https://www.trulens.org/ Trulens], [https://arize.com/ Arize AI] or [https://github.com/confident-ai/deepeval DeepEval].
 To evaluate a RAG application, it is important to consider both the retrieval and generation components of the system. The retrieval component should be able to find relevant information from the knowledge source quickly and efficiently. The generation component should be able to use this information to produce accurate and informative responses.

← Older revision		Revision as of 10:39, 26 September 2023
Line 90:		Line 90:
	= RAG Application Evaluation =		= RAG Application Evaluation =
	* [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag?utm_source=post-email-title&publication_id=1643556&post_id=137378927 LlamaIndex: How to evaluate your RAG applications \| Ryan Nguyen - How AI Built This] ... There are a few tools in the market that can help you to have a deep evaluation of your app such as [https://www.trulens.org/ Trulens], [https://arize.com/ Arize AI] or [https://github.com/confident-ai/deepeval DeepEval].		* [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag?utm_source=post-email-title&publication_id=1643556&post_id=137378927 LlamaIndex: How to evaluate your RAG applications \| Ryan Nguyen - How AI Built This] ... There are a few tools in the market that can help you to have a deep evaluation of your app such as [https://www.trulens.org/ Trulens], [https://arize.com/ Arize AI] or [https://github.com/confident-ai/deepeval DeepEval].
		+
		+	To evaluate a RAG application, it is important to consider both the retrieval and generation components of the system. The retrieval component should be able to find relevant information from the knowledge source quickly and efficiently. The generation component should be able to use this information to produce accurate and informative responses.
		+
		+	There are a number of different metrics that can be used to evaluate RAG applications. Some common metrics include:
		+
		+	* Faithfulness: This metric measures how well the generated response is supported by the retrieved information.
		+	* Answer relevancy: This metric measures how relevant the generated response is to the original prompt.
		+	* Context relevancy: This metric measures how relevant the retrieved information is to the original prompt.
		+	* Context recall: This metric measures how much of the relevant information from the knowledge source was retrieved.
		+
		+	In addition to these quantitative metrics, it is also important to evaluate the quality of the generated responses qualitatively. This can be done by having human evaluators assess the responses for accuracy, informativeness, and fluency. One way to evaluate a RAG application is to use a held-out test set. This test set should consist of a set of prompts and ground truth responses. The RAG application can then be evaluated on its ability to generate responses that are similar to the ground truth responses. Another way to evaluate a RAG application is to use a user-study. In a user-study, users are given a set of tasks to complete using the RAG application. The users' performance on these tasks can then be used to evaluate the effectiveness of the application. Overall, the evaluation of RAG applications is a complex task that requires the consideration of a number of different factors. However, by using a combination of quantitative and qualitative metrics, it is possible to get a good understanding of the performance of a RAG application.
		+
		+	Here are some additional tips for evaluating RAG applications:
		+
		+	* Use a variety of metrics to assess the performance of the application. This will help you to get a more complete picture of the application's strengths and weaknesses.
		+	* Use a held-out test set to evaluate the application on data that it has not seen before. This will help you to get a more accurate estimate of the application's performance in the real world.
		+	* Consider using a user-study to evaluate the application from the user's perspective. This can help you to identify areas where the application can be improved to make it more user-friendly and effective.

@@ Line 22: / Line 22: @@
 * [[Embedding]] ... [[Fine-tuning]] ... [[Retrieval-Augmented Generation (RAG)|RAG]] ... [[Agents#AI-Powered Search|Search]] ... [[Clustering]] ... [[Recommendation]] ... [[Anomaly Detection]] ... [[Classification]] ... [[Dimensional Reduction]].  [[...find outliers]]
 * [https://howaibuildthis.substack.com/p/so-you-want-to-improve-your-rag-pipeline So you want to improve your RAG pipeline | Ryan Nguyen - How AI Built This] ... Ways to go from prototype to production with LlamaIndex
-* [https://howaibuildthis.substack.com/p/llamaindex-how-to-evaluate-your-rag?utm_source=post-email-title&publication_id=1643556&post_id=137378927 LlamaIndex: How to evaluate your RAG applications | Ryan Nguyen - How AI Built This] ... There are a few tools in the market that can help you to have a deep evaluation of your app such as [https://www.trulens.org/ Trulens], [https://arize.com/ Arize AI] or [https://github.com/confident-ai/deepeval DeepEval].
 Retrieval-Augmented Generation (RAG) is a method of generating text by first retrieving relevant information from a knowledge base and then generating text based on that information. This can be done by using a retrieval model to find relevant documents or passages, and then using a language model to generate text that is consistent with the retrieved information.
@@ Line 87: / Line 87: @@
      </tr>
 </table>