Difference between revisions of "Agents/Assistants"

From
Jump to: navigation, search
m (Solid | Inrupt)
m
 
(9 intermediate revisions by the same user not shown)
Line 13: Line 13:
 
[https://www.bing.com/news/search?q=software+agent+ai++assistant&qft=interval%3d%228%22 ...Bing News]  
 
[https://www.bing.com/news/search?q=software+agent+ai++assistant&qft=interval%3d%228%22 ...Bing News]  
  
* [[Agents/Assistants]] ... [[Robotic Process Automation (RPA)|Robotic Process Automation]] ... [[Personal Companions]] ... [[Personal Productivity|Productivity]] ... [[Email]] ... [[Negotiation]] ... [[LangChain]]
+
* [[Agents/Assistants]] ... [[Robotic Process Automation (RPA)|Robotic Process Automation]] ... [[Personal Companions]] ... [[Personal Productivity|Productivity]] ... [[Email]] ... [[Negotiation]] ... [[LangChain]] ... [[Memory]]
 
* [[Development]] ... [[Notebooks]] ... [[Development#AI Pair Programming Tools|AI Pair Programming]] ... [[Codeless Options, Code Generators, Drag n' Drop|Codeless, Generators, Drag n' Drop]] ... [[Hugging Face]] ... [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] ... [[Platforms: AI/Machine Learning as a Service (AIaaS/MLaaS)|AIaaS/MLaaS]]
 
* [[Development]] ... [[Notebooks]] ... [[Development#AI Pair Programming Tools|AI Pair Programming]] ... [[Codeless Options, Code Generators, Drag n' Drop|Codeless, Generators, Drag n' Drop]] ... [[Hugging Face]] ... [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] ... [[Platforms: AI/Machine Learning as a Service (AIaaS/MLaaS)|AIaaS/MLaaS]]
 
* [[Immersive Reality]] ... [[Metaverse]] ... [[Omniverse]] ... [[Transhumanism]] ... [[Religion]]  
 
* [[Immersive Reality]] ... [[Metaverse]] ... [[Omniverse]] ... [[Transhumanism]] ... [[Religion]]  
 
* [[Simulation]] ... [[Simulated Environment Learning]] ... [[World Models]] ... [[Minecraft]]: [[Minecraft#Voyager|Voyager]] (an AI agent powered by an LLM introduced to the world of Minecraft)
 
* [[Simulation]] ... [[Simulated Environment Learning]] ... [[World Models]] ... [[Minecraft]]: [[Minecraft#Voyager|Voyager]] (an AI agent powered by an LLM introduced to the world of Minecraft)
* [[Telecommunications]] ... [[Computer Networks]] ... [[Telecommunications#5G|5G]] ... [[Satellite#Satellite Communications|Satellite Communications]] ... [[Quantum Communications]] ... [[Agents#Communication | Communication Agents]] ... [[Smart Cities]] ...  [[Digital Twin]] ... [[Internet of Things (IoT)]]
+
* [[Telecommunications]] ... [[Computer Networks]] ... [[Telecommunications#5G|5G]] ... [[Satellite#Satellite Communications|Satellite Communications]] ... [[Quantum Communications]] ... [[Smart Cities]] ...  [[Digital Twin]] ... [[Internet of Things (IoT)]] ... [[AI Generated Broadcast Content|AI Broadcast; Radio, Stream, TV]] ... [[Computer_Networks#Space-based Data Centers|Space-based Data Centers]]
* [[Mixture-of-Experts (MoE)]] ... [[Mistral]] ... [[Chain of Thought (CoT)]]
 
* [[Memory]] ... [[Memory Networks]] ... [[Hierarchical Temporal Memory (HTM)]] ... [[Lifelong Learning]]
 
 
* [[Artificial General Intelligence (AGI) to Singularity]] ... [[Inside Out - Curious Optimistic Reasoning| Curious Reasoning]] ... [[Emergence]] ... [[Moonshots]] ... [[Explainable / Interpretable AI|Explainable AI]] ...  [[Algorithm Administration#Automated Learning|Automated Learning]]
 
* [[Artificial General Intelligence (AGI) to Singularity]] ... [[Inside Out - Curious Optimistic Reasoning| Curious Reasoning]] ... [[Emergence]] ... [[Moonshots]] ... [[Explainable / Interpretable AI|Explainable AI]] ...  [[Algorithm Administration#Automated Learning|Automated Learning]]
 
* [[Python]] ... [[Generative AI with Python|GenAI w/ Python]] ... [[JavaScript]] ... [[Generative AI with JavaScript|GenAI w/ JavaScript]] ... [[TensorFlow]] ... [[PyTorch]]
 
* [[Python]] ... [[Generative AI with Python|GenAI w/ Python]] ... [[JavaScript]] ... [[Generative AI with JavaScript|GenAI w/ JavaScript]] ... [[TensorFlow]] ... [[PyTorch]]
Line 26: Line 24:
 
* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Grok]] | [https://x.ai/ xAI] ... [[Groq]] ... [[Ernie]] | [[Baidu]] ... [[DeepSeek]] ...  [[Alibaba]]
 
* [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Grok]] | [https://x.ai/ xAI] ... [[Groq]] ... [[Ernie]] | [[Baidu]] ... [[DeepSeek]] ...  [[Alibaba]]
 
** [[ChatGPT#Integration| ChatGPT Integration]]
 
** [[ChatGPT#Integration| ChatGPT Integration]]
* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]]  ...[[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
+
* [[Large Language Model (LLM)]] ... [[Large Language Model (LLM)#Multimodal|Multimodal]] ... [[Foundation Models (FM)]] ... [[Generative Pre-trained Transformer (GPT)|Generative Pre-trained]] ... [[Transformer]] ... [[Attention]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]]
* [[Attention]] Mechanism  ...[[Transformer]] ...[[Generative Pre-trained Transformer (GPT)]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]]
 
 
* [[Policy]]  ... [[Policy vs Plan]] ... [[Constitutional AI]] ... [[Trust Region Policy Optimization (TRPO)]] ... [[Policy Gradient (PG)]] ... [[Proximal Policy Optimization (PPO)]]
 
* [[Policy]]  ... [[Policy vs Plan]] ... [[Constitutional AI]] ... [[Trust Region Policy Optimization (TRPO)]] ... [[Policy Gradient (PG)]] ... [[Proximal Policy Optimization (PPO)]]
 
* [[Multi-Task Learning (MTL)]] ... [[SMART - Multi-Task Deep Neural Networks (MT-DNN)]]
 
* [[Multi-Task Learning (MTL)]] ... [[SMART - Multi-Task Deep Neural Networks (MT-DNN)]]
Line 37: Line 34:
 
* [[Case Studies]]
 
* [[Case Studies]]
 
** [[Human Resources (HR)]] ... [[Toys]] ... [[Education]] ... [[Psychology - Mental Health]] ... [[AI Generated Broadcast Content#Television (TV)|Television (TV)]] ... [[Writing/Publishing]] ... [[Marketing]] ... [[Healthcare]] ... [[Real Estate]] ... [[Development]] ... [[Law]] ... [[Politics]] ... [[Strategy & Tactics]] ... [[Travel & Tourism]]
 
** [[Human Resources (HR)]] ... [[Toys]] ... [[Education]] ... [[Psychology - Mental Health]] ... [[AI Generated Broadcast Content#Television (TV)|Television (TV)]] ... [[Writing/Publishing]] ... [[Marketing]] ... [[Healthcare]] ... [[Real Estate]] ... [[Development]] ... [[Law]] ... [[Politics]] ... [[Strategy & Tactics]] ... [[Travel & Tourism]]
* [[Bot Framework]]
 
  
 
= Defining Chatbots, Assistants, and Agents =
 
= Defining Chatbots, Assistants, and Agents =
Line 217: Line 213:
  
 
<img src="https://miro.medium.com/v2/resize:fit:828/format:webp/1*iS4cHbQAXv-43Pi1EJxK8A.png" width="800">
 
<img src="https://miro.medium.com/v2/resize:fit:828/format:webp/1*iS4cHbQAXv-43Pi1EJxK8A.png" width="800">
 
+
<youtube>r5JvhWpNbqM</youtube>
 
<youtube>6E-x0HY_ZVI</youtube>
 
<youtube>6E-x0HY_ZVI</youtube>
 
<youtube>PfY9lVtM_H0</youtube>
 
<youtube>PfY9lVtM_H0</youtube>
Line 261: Line 257:
 
<youtube>yHouKDbiPPs</youtube>
 
<youtube>yHouKDbiPPs</youtube>
 
<youtube>NJ4N2n8nb8w</youtube>
 
<youtube>NJ4N2n8nb8w</youtube>
 +
<youtube>5SgJKZLBrmg</youtube>
  
 
== Autonomous GPT ==
 
== Autonomous GPT ==
Line 300: Line 297:
 
AutoGen Studio 2.0 simplifies AI development by providing an intuitive interface for multi-agent workflows. The Build section lets you create agents and define their skills. The Playground section lets you test and observe their behavior in real time, and the Python API gives developers detailed control over the workflow.
 
AutoGen Studio 2.0 simplifies AI development by providing an intuitive interface for multi-agent workflows. The Build section lets you create agents and define their skills. The Playground section lets you test and observe their behavior in real time, and the Python API gives developers detailed control over the workflow.
  
 +
<youtube>byPbxEH5V8E</youtube>
 
<youtube>4ZqJSfV4818</youtube>
 
<youtube>4ZqJSfV4818</youtube>
 
<youtube>Cl19yWHhc2g</youtube>
 
<youtube>Cl19yWHhc2g</youtube>
Line 353: Line 351:
  
 
Solid is a technology for organizing data, applications, and identities on the web, championed by Tim Berners-Lee. It gives you complete control over your data, allowing you to run your own personal artificial intelligence assistant securely instead of handing your data to a tech giant.
 
Solid is a technology for organizing data, applications, and identities on the web, championed by Tim Berners-Lee. It gives you complete control over your data, allowing you to run your own personal artificial intelligence assistant securely instead of handing your data to a tech giant.
 
 
<youtube>DBB_LMepuQ4</youtube>
 
<youtube>DBB_LMepuQ4</youtube>
 
<youtube>Ay7GSLX9tHM</youtube>
 
<youtube>Ay7GSLX9tHM</youtube>

Latest revision as of 10:57, 15 September 2026

YouTube ... Quora ...Google search ...Google News ...Bing News

Defining Chatbots, Assistants, and Agents

The tech world started using the word "Agent" because they ruined the word "assistant." Over the last decade, companies watered down the term to describe basic smart speakers and text-only chatbots that simply fetch links or set timers. Because those tools are so passive, engineers needed a new label to describe systems capable of seeing, reasoning, and acting autonomously.

The easiest way to differentiate between them is to look at their capabilities.

  • Chatbot: Designed to handle basic communication tasks. It responds to questions with pre-approved answers using scripted decision trees. When you enter a query, the chatbot picks up keywords and responds with a scripted response from its knowledge base. In many cases, chatbots require you to choose responses from set phrases (like choosing between ‘account’, ‘order’, or ‘billing’), which severely limits what you can ask.
  • Assistant: Reactive but conversational. Instead of relying purely on scripted answers, an Assistant uses Machine Learning and Natural Language Processing to understand your intent and respond in a personalized way. It remembers previous interactions and handles a wider variety of tasks.
  • Personal Companion: Proactive and reactive. It builds an ongoing relationship with the user.
  • Agent: Proactive. It performs multi-step tasks and makes decisions on your behalf.

An intelligent agent perceives its environment, takes actions autonomously to achieve goals, and improves its performance by acquiring knowledge. An agent has an "objective function" encapsulating all its goals. It designs and executes whatever plan maximizes the expected value of that function. For example, a reinforcement learning agent uses a "reward function" to shape desired behavior, while an evolutionary algorithm uses a "fitness function." (Source: Wikipedia).


Agents flip our copilot paradigm: from AI being human's copilot to humans being AI's copilot.


The paradigm of AI copilots is undergoing a transformation. We are shifting from AI serving as a supportive assistant to humans taking on the role of AI copilots. This enables you to harness the power of AI to enhance decision-making and efficiency. It signifies a move towards a more collaborative relationship where humans actively engage with AI to augment their own productivity.

The 3 Stages of AI (Steven Bathiche)

  1. Beside: AI acts as a valuable tool, a helper, or a sidebar. It is an assistant that aids with research, writing, and presentations. It breaks our mental blocks and manages personal scheduling and finances. By automating data entry and email processing, it frees up time for more important work.
  2. Inside: AI becomes deeply integrated into our daily lives, understanding our preferences and anticipating our actions. Instead of waiting for you to make a keystroke or select an application, the AI takes the lead. It chooses the best app for you and coaches you through the possibilities.
  3. Outside: AI agents act as independent intermediaries. They pull information from disparate data sources and interpret the world for us. They help navigate context and act as facilitators, completely eliminating the challenge of starting with a blank sheet.

Why go to a search service if assistants are built into every application you use?


How an Agent Works

A true assistant is a trusted, fully capable partner that does not just retrieve information. It manages complexity.

Imagine a single application running silently in the background of your devices that merges deep coding logic, instant voice dictation, emotional conversation, and complete operating system control. The agent operates on a continuous feedback loop. It perceives its environment by taking a screenshot, decides on a plan, takes an action by clicking a button, verifies if the action worked by taking another screenshot, and repeats until the goal is met. If a website layout changes or an error pop-up appears, the agent sees the obstacle in the next frame and adjusts its plan. It does not just tell you how to do something; it does the actual work. It functions less like a digital butler and more like a synthetic colleague sitting next to you, sharing your keyboard and monitor.

  • The Voice (Conversational UI): The layer that keeps you in the loop, letting you verbally interrupt, correct, or guide the system in real time.
  • The Eyes (Screen Vision and Camera): The system continuously captures your screen state or live video feeds. It understands UI layouts, spatial relationships, and visual context that text alone cannot convey.
  • The Brain (Reasoning Engine): Models evaluate the visual input against your goal to determine the next logical step.
  • The Hands (OS Automation and Dictation): Tools execute the plan by taking over the mouse, clicking buttons, or typing perfectly formatted text directly into your active window.

Browser Use and Computer Use

When we talk about how AI agents actually get things done on your device, we generally divide their physical abilities into two categories.

Browser Use means the AI agent navigates the internet exactly like a human does. It does not just quietly ping a server behind the scenes. It literally opens a web browser, types in a URL, reads the webpage, clicks on links, and fills out forms.

  • The Analogy: Think of it like hiring a virtual assistant to sit at their own computer and surf the web for you.
  • Example: You ask an agent to find highly rated local restaurants. The browser use agent opens a maps website, searches the area, reads the reviews right off the screen, and extracts the top spots into a neat list for you.

Computer Use goes a massive step further. Instead of being restricted to a web browser, a computer use agent has control over your entire operating system. It can see your active screen, move your mouse cursor, double-click local desktop icons, open file folders, and type on a virtual keyboard.

  • The Analogy: It is exactly like letting an IT support tech take temporary control of your screen via remote desktop.
  • Example: You tell your agent to zip up all the latest design files for a project and email them to a coworker. The computer use agent physically moves the mouse to your documents folder, highlights the files, right-clicks to compress them, opens your desktop email app, types the email address, attaches the folder, and clicks send.

Recursive Self-Improvement (RSI)

RSI stands for Recursive Self-Improvement. It describes an AI system that helps research, code, train, or evaluate its own next generation. Think of it as an AI building a smarter version of itself. Once that smarter successor comes online, it builds an even more capable system. This creates a compounding feedback loop where the AI's capabilities accelerate rapidly, often moving much faster than human safety checks and oversight can keep up.

J.A.R.V.I.S. (Just A Rather Very Intelligent System)

J.A.R.V.I.S. is the ultimate AI standard assistant created by Tony Stark in the Marvel Cinematic Universe. In Marvel Comics lore and the TV series Agent Carter, Edwin Jarvis was originally Tony Stark's human butler. Voiced by Paul Bettany, the AI appears in:

  • Iron Man (2008)
  • Iron Man 2 (2010)
  • The Avengers (2012)
  • Iron Man 3 (2013)
  • Avengers: Age of Ultron (2015), where his core code is ultimately uploaded into the synthetic body that becomes Vision.

Capability Requirements

Modern assistants now leverage advanced models to bridge the gap between intent and action. Often operating on-device for improved privacy and lower latency, a fully developed AI Assistant should possess the following capabilities:

  • Creative Inspiration: It should spark creativity by providing suggestions, ideas, and inspiration for artistic, writing, or design endeavors to help you overcome creative blocks.
  • Contextual Awareness: It needs to understand the context of a conversation, remember past interactions, and provide relevant, personalized responses.
    • Contextual Adaptation: It must dynamically adjust its behavior based on your specific environment and preferences. It should seamlessly switch tasks, like providing weather updates while managing a navigation request.
    • Natural Language Understanding (NLU): It must interpret commands and conversations accurately, handling complex sentence structures and context-dependent meanings.
    • Personalized Recommendations: It should suggest movies, books, or restaurants tailored to your specific interests based on past interactions.
    • Chained Recursive Capability: It needs to generate responses involving multiple levels of reasoning, handling complex queries through in-depth and nuanced discussions.
  • Continuous Learning: It must learn and improve over time. It requires memory to remember your preferences and refine its recommendations based on your direct feedback. It should also offer customization features so you can personalize its behavior and appearance.
  • Multimodal Interaction: It must support voice, text, and gestures so you can communicate however you prefer.
    • Voice Input and Output: It should understand voice commands accurately and generate natural verbal responses.
  • Knowledge Base Integration: It needs access to a comprehensive, up-to-date knowledge base to answer factual questions accurately.
    • Web Search Capability: It should search the live web to retrieve relevant information from reputable sources.
  • Entertainment:
    • It should suggest books, articles, or videos that align with your interests.
    • It should read stories aloud engagingly, adapting its tone based on the genre.
    • It should generate original stories collaboratively with you, offering writing prompts or feedback. It should support interactive storytelling where you make choices that influence the outcome.
    • It must possess literary analysis capabilities to discuss themes, characters, and writing styles.
    • Music Playback: It should access your music library, support streaming services, and organize playlists on demand.
    • Game Playing: It needs the ability to play strategy games like chess. It should remember your past tactics, adapt its gameplay to provide a challenge, and maintain a win-loss scoring record.
    • Joke Telling: It should have a repertoire of jokes across categories to provide a laugh when requested.
  • Travel Planning and Assistance: It should help organize travel arrangements, flights, accommodations, and offer real-time updates.
    • Mapping and Navigation: It needs map data to offer directions and real-time traffic updates.
  • List and Task Management: It must remember and organize shopping lists, item lists, and daily tasks. It should allow you to add items, set due dates, categorize, and prioritize effortlessly.
  • Social Interaction Enhancement: It should offer conversation prompts or etiquette guidance for networking scenarios.
    • Email and Messaging: It must integrate with messaging platforms so you can compose, check, and manage your inbox using voice or text.
    • Virtual Meeting Support: It should schedule meetings, send invitations, and provide meeting resources.
    • Calendar Management: It must add, edit, and organize events while sending reminders for important dates.
  • Integration with Smart Home Devices: It must connect to IoT devices, allowing you to control lights, thermostats, and cameras via voice.
  • Personalized News and Updates: It should gather news from various sources and present a concise summary tailored to your interests.
  • Financial Management: It needs to track expenses, set budgets, and provide basic financial insights.
  • Health and Growth:
    • It should support personal growth by providing guidance on self-improvement and skill development.
    • Personal Safety: It must provide emergency contact info, location sharing, and assistance in a crisis.
    • Well-being Support: It should remind you to exercise, meditate, hydrate, and maintain a healthy lifestyle. It should integrate seamlessly with fitness trackers.
    • Cognitive Enhancements: It should offer memory training or brain teasers to improve mental agility.
    • Emotional Intelligence: It needs to recognize human emotions, display empathy, and adapt its tone accordingly.
    • Educational Support: It should act as a tutor, offering explanations and assisting with research across subjects.
  • Shared Assistants: It should establish relationships with other users' AI to facilitate quick learning and discover common interests.
    • Similar to sharing a calendar, sharing knowledge between assistants fosters collective intelligence. This allows you to gain broader perspectives, access diverse experiences, and solve challenges collectively as a community.
  • Reliability: It must be highly responsive with minimal downtime, handling errors and interruptions gracefully.
    • Offline Functionality: It should perform basic tasks and provide core information even when the internet is disconnected.
  • Security and Privacy: It must incorporate robust security measures to protect your data using encryption and authentication. It should only collect necessary data, explicitly seek your consent, and provide transparent privacy settings. It must also detect and prevent malicious activities like phishing.
  • Accessibility and Availability: It must be designed for users with different abilities, offering text-to-speech and visual cues.
    • Third-Party Integration: It needs to connect to external apps to order food, book rides, or make reservations.
    • Multi-device Compatibility: It should work across smartphones, tablets, smart speakers, wearables, and immersive VR/AR devices.
    • Multilingual Support: It must accurately understand and respond in multiple languages for a localized, inclusive experience.
  • Affordability: It must be priced reasonably so the cost aligns with the value it provides.

Agent Frameworks and Platforms

OpenClaw and Related Efforts

OpenClaw is an open-source autonomous AI agent framework capable of executing multi-step tasks across various digital platforms. Created in late 2025 by Austrian developer Peter Steinberger, the software runs locally on your machine and serves as a bridge between Large Language Models and local system tools. Unlike traditional chatbots that only generate text, OpenClaw performs active workflows like managing files, sending emails, and writing code. You initiate these tasks via messaging apps like WhatsApp, Telegram, and Slack.

The project gained viral popularity in early 2026. People often described it as "AI that actually does things" because it retains persistent memory and executes complex chains of actions without needing continuous human oversight. However, its deep system access drew scrutiny from cybersecurity researchers regarding potential unauthorized data modification.

Following OpenClaw's rapid rise, major technology companies accelerated their own competing autonomous agents:

  • Google: In May 2026, reports emerged that Google was internally testing a competitor codenamed "Remy". Integrated into a staff-only version of Gemini, Remy is described as a 24/7 personal agent capable of proactively handling workflows across the Google Workspace ecosystem (Gmail, Calendar, Drive).
  • OpenAI: Instead of releasing a direct clone, OpenAI hired OpenClaw creator Peter Steinberger in February 2026 to work on their next generation of personal agents.
  • Microsoft: Microsoft developed an enterprise-focused alternative codenamed "Project Lobster." This evolved into the official product Microsoft Scout, announced in June 2026. Scout integrates into Microsoft 365 and Windows, offering autonomous scheduling and data processing wrapped in enterprise-grade security (Entra ID).
  • Anthropic: Anthropic is reportedly developing a remote control feature called "Dispatch" to allow its models to operate agents in the background.
  • Meta: Meta launched a desktop application called Manus to bring agentic capabilities directly to personal computers. Meta also acquired Moltbook, a viral social media platform populated exclusively by AI agents.
  • Other Competitors: Salesforce is developing an enterprise alternative named "Albert." Perplexity launched a local agent system dubbed "Personal Computer." The open-source community spawned forks like NanoClaw, Hermes Agent, and ZeroClaw.

LlamaIndex’s Agent

LlamaIndex's Agent is an LLM chatbot developed by LlamaIndex, a company specializing in alpaca and llama products. The agent is trained on a massive dataset of text and code, including their own database. It can answer questions about alpaca biology and care, generate creative text formats (poems, scripts, emails), and translate languages.

LangChain’s Agent

LangChain's Agent automates tasks by connecting an LLM to external data. It understands natural language to answer questions comprehensively, generate creative text, translate languages, and automate tasks that normally require human intervention. For example, a customer service chatbot can use LangChain to answer complex questions accurately, or a software team can use it to write unit tests automatically.

DeepMind's WebAgent

Google DeepMind's WebAgent can autonomously navigate and interact with websites. You give it natural language instructions, and it completes the tasks.

Key Features:

  • Planning: It breaks down complex instructions into manageable steps and analyzes lengthy HTML documents.
  • Dual LLM Design: It uses HTML-T5 (specializing in understanding website layouts) and Flan-U-PaLM (capable of generating executable Python code).
  • Program Synthesis: It translates your instructions into executable Python code, allowing it to directly interact with websites, fill forms, click buttons, and extract data.
  • Reinforcement Learning: It learns from experience and improves over time.

Jarvis (Microsoft)

Jarvis is a project from Microsoft that uses ChatGPT as the project controller. When you make a request, Jarvis plans the task and chooses which models it needs from the Hugging Face community. It has those models perform the tasks, integrates their predictions, and generates a final response for you.


HuggingGPT is one instance of Jarvis, a web-based chatbot at Hugging Face, an online AI community which hosts thousands of open-source models.


Auto-GPT

Auto-GPT is an open-source Python application based on GPT-4 that can self-prompt. If you state an end goal, the system works out the steps and carries them out. It does all the follow-up work for you, asking and answering its own prompts in a loop. It manages memory by reading from databases, performs web searches, and interacts with APIs unattended.


Auto-GPT is not yet capable of achieving the AGI (Artificial General Intelligence) due to data quality, generalization, and explainability issues.


The actions of the agents are visible to you in the form of:

  • Thoughts: The agent shares its thoughts after completing an action.
  • Reasoning: It explains why it chose a particular course of action.
  • Plan: It lists the new set of tasks.
  • Criticism: It critically reviews its choices by identifying limitations.

Reflexion

Reflexion gives an agent dynamic memory and self-reflection capabilities. It allows the agent to learn from its mistakes and solve novel problems efficiently through trial and error. It uses a heuristic that helps the agent pinpoint hallucinations, avoid repeating the same failed actions, and build an internal memory map of its environment.


Self-reflection allows humans to efficiently solve novel problems through a process of trial and error.


Autonomous GPT

AgentGPT

AgentGPT allows you to configure and deploy Autonomous AI agents directly in your browser using LangChain and Auto-GPT. You name your custom AI, give it a goal, and it will attempt to reach that goal by thinking of tasks, executing them, and learning from the results.

BabyAGI

OpenAGI

OpenAGI is an open-source research platform designed to offer complex, multi-step tasks. It integrates standard benchmark tasks for evaluation alongside open-ended tasks that require creative problem-solving via plugins and APIs.

AutoGen

AutoGen Studio 2.0 simplifies AI development by providing an intuitive interface for multi-agent workflows. The Build section lets you create agents and define their skills. The Playground section lets you test and observe their behavior in real time, and the Python API gives developers detailed control over the workflow.

MultiOn

MultiOn acts as a personal life copilot that uses the browser to execute complex tasks. It is a web browser (and a ChatGPT plugin) that interacts with anything on the internet on your behalf, like posting on social media or ordering products online.

CrewAI

CrewAI is a framework designed for orchestrating role-playing autonomous agents. It empowers agents to assume specific roles, share goals, and operate together as a cohesive unit. Whether you want to build an automated customer service ensemble or a multi-agent research team, CrewAI provides the backbone for sophisticated interaction.

Commercial AI Assistants

Conversational AI Market Leaders

The market is currently defined by powerful multimodal models that serve as the brains for modern assistants:

  • OpenAI: The o1 and o3 models focus heavily on deep reasoning and complex problem-solving.
  • Google: Gemini 2.0 emphasizes native multimodal integration and proactive mobile assistance.
  • Anthropic: Claude 3.5 and 3.7 are known for high-fidelity reasoning and nuanced professional workflows.

Google Surfaces

Google's AI assistant combines features from DeepMind and Google Bard. The Smart Google Expert (SGE) uses the RT-2 model to understand visual and linguistic cues. Google Assistant controls your smart home devices, answers questions, plays music, and manages everyday tasks.

Duplex

Duplex is a technology that allows Google Assistant to make phone calls naturally. It uses machine learning to mimic human speech patterns. This allows it to call businesses and make reservations or appointments without the person on the other end realizing they are talking to a machine.

Duet

Google Duet is an AI collaborator built into Google Workspace. It listens to your meetings, automatically generates notes, and produces summaries with action items. It also creates custom backgrounds for video calls, analyzes data in Sheets to find patterns, and writes creative content.

Alexa and Lex

With Alexa, developers build natural voice experiences that offer a more intuitive way to interact with technology. Through a voice user interface, services like Alexa communicate effortlessly, solve problems, and get smarter over time. Lex is the underlying technology that allows developers to build their own conversational bots.

Solid | Inrupt

Solid is a technology for organizing data, applications, and identities on the web, championed by Tim Berners-Lee. It gives you complete control over your data, allowing you to run your own personal artificial intelligence assistant securely instead of handing your data to a tech giant.

Lindy

Lindy is an AI assistant that helps with daily tasks. It manages your calendar, drafts personalized email replies in your voice, automatically handles scheduling conflicts, and summarizes podcasts or YouTube videos for you.

Architecture and Development

Hybrid Assistants

A "hybrid assistant" (or a model-of-models architecture) links a friendly conversational interface with specialized background agents. If the front-end assistant cannot answer your query, it smoothly transfers the conversation to a specialized software agent or a live human. The assistant acts as the interface for you, receiving inputs and providing outputs, while the background agent performs the complex actions.

Prompt Engineering for Assistants

Customizing AI assistants for professional workflows relies heavily on Prompt Engineering. By utilizing system instructions and persona-tuning, developers constrain the model's behavior, tone, and domain expertise. This ensures the assistant acts as a specialized expert (like a legal analyst) rather than a general-purpose chatbot.

Application Programming Interface (API)

Large Language Models use APIs to connect to external tools and data:

  • Google Cloud Natural Language API: Analyzes text to perform sentiment analysis and entity recognition.
  • IBM Watson NLU API: Extracts insights like emotion and keywords from text.
  • OpenAI API: Performs tasks like language translation and powers the Assistants API, which includes a Code Interpreter to write and run Python code in a sandbox.
  • Twitter API: Allows models to access real-time tweets for topic modeling.
  • Wikipedia API: Gives models access to vast amounts of factual knowledge to improve question-answering.

Building ChatBots

Are You Ready for Your Own AI Assistant?

Imagine having a personal assistant who is always there to help you. It can answer your questions, set reminders, and control your smart home devices. Virtual assistants save you time, money, and stress. If you are looking for a way to improve your productivity, you first need to decide what tasks you want to hand off. Once you know what you need help with, you can explore the tools above to find the perfect fit. Setting up the right assistant means you get more done in less time, freeing you up to actually relax and enjoy your life.