Constitutional AI - Revision history

BPeat at 13:55, 23 March 2024

2024-03-23T13:55:13Z

BPeat: Text replacement - "* Conversational AI ... ChatGPT | OpenAI ... Bing | Microsoft ... Bard | Google ... Claude | Anthropic ... Perplexity ... You ... Ernie | Baidu" to "* Conversational AI ... [[C...

2024-03-16T16:17:27Z

BPeat at 01:49, 3 September 2023

2023-09-03T01:49:30Z

BPeat at 22:22, 12 July 2023

2023-07-12T22:22:55Z

BPeat at 18:54, 7 July 2023

2023-07-07T18:54:56Z

BPeat at 18:52, 7 July 2023

2023-07-07T18:52:45Z

BPeat at 16:45, 30 June 2023

2023-06-30T16:45:50Z

BPeat at 02:40, 17 June 2023

2023-06-17T02:40:59Z

BPeat at 02:32, 17 June 2023

2023-06-17T02:32:06Z

BPeat at 01:41, 17 June 2023

2023-06-17T01:41:12Z

@@ Line 22: / Line 22: @@
 * [[Policy]] ... [[Policy vs Plan]] ... [[Constitutional AI]] ... [[Trust Region Policy Optimization (TRPO)]] ... [[Policy Gradient (PG)]] ... [[Proximal Policy Optimization (PPO)]]
 * [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
-* [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
+* [[Agents]] ... [[Robotic Process Automation (RPA)|Robotic Process Automation]] ... [[Assistants]] ... [[Personal Companions]] ... [[Personal Productivity|Productivity]] ... [[Email]] ... [[Negotiation]] ... [[LangChain]]
 * [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Generative AI]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
 * [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Ernie]] | [[Baidu]]

@@ Line 24: / Line 24: @@
 * [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
 * [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Generative AI]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
-* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing]] | [[Microsoft]] ... [[Bard]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[Ernie]] | [[Baidu]]
+* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Ernie]] | [[Baidu]]
 * [[Claude]] | [https://www.anthropic.com/ Anthropic]
 * [[Reinforcement Learning (RL) from Human Feedback (RLHF)]]

@@ Line 23: / Line 23: @@
 * [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
 * [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
-* [[Generative AI]] ... [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing]] | [[Microsoft]] ... [[Bard]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[Ernie]] | [[Baidu]]
+* [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Generative AI]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
 * [[Claude]] | [https://www.anthropic.com/ Anthropic]
 * [[Reinforcement Learning (RL) from Human Feedback (RLHF)]]

@@ Line 23: / Line 23: @@
 * [[What is Artificial Intelligence (AI)? | Artificial Intelligence (AI)]] ... [[Machine Learning (ML)]] ... [[Deep Learning]] ... [[Neural Network]] ... [[Reinforcement Learning (RL)|Reinforcement]] ... [[Learning Techniques]]
 * [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
-* [[Generative AI]]  ... [[Conversational AI]] ... [[OpenAI]]'s [[ChatGPT]] ... [[Perplexity]]  ... [[Microsoft]]'s [[Bing]] ... [[You]] ...[[Google]]'s [[Bard]] ... [[Baidu]]'s [[Ernie]]
+* [[Generative AI]] ... [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing]] | [[Microsoft]] ... [[Bard]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[Ernie]] | [[Baidu]]
 * [[Claude]] | [https://www.anthropic.com/ Anthropic]
 * [[Reinforcement Learning (RL) from Human Feedback (RLHF)]]

← Older revision		Revision as of 18:54, 7 July 2023
Line 28:		Line 28:
	* [https://medium.com/mlearning-ai/paper-review-constituional-ai-training-llms-using-principles-16c68cfffaef Paper Review: Constitutional AI, Training LLMs using Principles]		* [https://medium.com/mlearning-ai/paper-review-constituional-ai-training-llms-using-principles-16c68cfffaef Paper Review: Constitutional AI, Training LLMs using Principles]

		+	Constitutional AI is a method for training AI systems using a set of rules or principles that act as a “constitution” for the AI system. This approach allows the AI system to operate within a societally accepted framework and aligns it with human intentions. Some benefits of using Constitutional AI include allowing a model to explain why it is refusing to provide an answer, improving transparency of AI decision making, and controlling AI behavior more precisely with fewer human labels.
		+	* Constitutional AI is a technique that aims to imbue systems with “values” defined by a “constitution”³.
		+	* This makes the behavior of systems both easier to understand and simpler to adjust as needed³.
		+	* The system uses a set of principles to make judgments about outputs, hence the term “Constitutional”⁴.
		+	* This approach makes the values of the AI system easier to understand and easier to adjust as needed⁴.

−	Constitutional AI is a method for training AI systems using a set of rules or principles that act as a “constitution” for the AI system. This approach allows the AI system to operate within a societally accepted framework and aligns it with human intentions1.
−
−	Some benefits of using Constitutional AI include allowing a model to explain why it is refusing to provide an answer, improving transparency of AI decision making, and controlling AI behavior more precisely with fewer human labels.

	<youtube>5GqtRXY-80k</youtube>		<youtube>5GqtRXY-80k</youtube>

@@ Line 2: / Line 2: @@
 |title=PRIMO.ai
 |titlemode=append
-|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, Tensorflow, Facebook, Meta, Google, Nvidia, Microsoft, Azure, Amazon, AWS
+|keywords=ChatGPT, artificial, intelligence, machine, learning, GPT-4, GPT-5, NLP, NLG, NLC, NLU, models, data, singularity, moonshot, Sentience, AGI, Emergence, Moonshot, Explainable, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Hugging Face, OpenAI, Tensorflow, OpenAI, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, LLM, metaverse, assistants, agents, digital twin, IoT, Transhumanism, Immersive Reality, Generative AI, Conversational AI, Perplexity, Bing, You, Bard, Ernie, prompt Engineering LangChain, Video/Image, Vision, End-to-End Speech, Synthesize Speech, Speech Recognition, Stanford, MIT |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
 |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
 }}
 [https://www.youtube.com/results?search_query=Constitutional+AI YouTube]
@@ Line 11: / Line 20: @@
 [https://www.bing.com/news/search?q=Constitutional+AI&qft=interval%3d%228%22 ...Bing News]
 * [[Reinforcement Learning (RL)]]
 * [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
@@ Line 16: / Line 26: @@
 * [[Claude]] | [https://www.anthropic.com/ Anthropic]
 * [[Reinforcement Learning (RL) from Human Feedback (RLHF)]]
 * [https://medium.com/mlearning-ai/paper-review-constituional-ai-training-llms-using-principles-16c68cfffaef Paper Review: Constitutional AI, Training LLMs using Principles]

@@ Line 14: / Line 14: @@
 * [[Assistants]] ... [[Personal Companions]] ... [[Agents]]  ... [[Negotiation]] ... [[LangChain]]
 * [[Generative AI]]  ... [[Conversational AI]] ... [[OpenAI]]'s [[ChatGPT]] ... [[Perplexity]]  ... [[Microsoft]]'s [[Bing]] ... [[You]] ...[[Google]]'s [[Bard]] ... [[Baidu]]'s [[Ernie]]
 * [[Reinforcement Learning (RL) from Human Feedback (RLHF)]]
 * [[Policy]]   ... [[Policy vs Plan]] ... [[Constitutional AI]] ... [[Trust Region Policy Optimization (TRPO)]] ... [[Policy Gradient (PG)]] ... [[Proximal Policy Optimization (PPO)]]
@@ Line 28: / Line 29: @@
 <youtube>fqC3D-zNJUM</youtube>
 == RL from AI Feedback' (RLAIF) ==
 It is a process that involves training a preference model from a dataset of AI preferences and then using that preference model as the reward signal for training with reinforcement learning. RLAIF is a method for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so the method is referred to as ‘Constitutional AI’. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase, an initial model is sampled from, then self-critiques and revisions are generated, and then the original model is finetuned on revised responses. In the [[Reinforcement Learning (RL)|RL]] phase, samples are taken from the finetuned model and a model is used to evaluate which of the two samples is better. A preference model is then trained from this dataset of AI preferences. The preference model is used as the reward signal for training with [[Reinforcement Learning (RL)|RL]].