Agent Development Framework Comparison

From PRIMO.ai
Jump to navigation Jump to search



Which framework or SDK should you use to build a custom AI agent?

Agent development frameworks and software development kits (SDKs) provide reusable programming components for applications that use AI models, access tools, retrieve information, maintain state, coordinate operations, and complete multi-step tasks. Unlike ready-to-use AI agents, these environments allow developers to design agentic behavior and integrate it into their own software.

This guide compares six principal development frameworks:

  • OpenAI Agents SDK — OpenAI
  • Claude Agent SDK — Anthropic
  • Google Agent Development Kit (ADK) — Google
  • LangGraph — LangChain
  • CrewAI — CrewAI
  • Qwen-Agent — Alibaba / Qwen

It also identifies managed agent platforms and specialized research APIs that may be preferable to operating an agent runtime yourself.

The objective is to select the simplest development approach that satisfies the application's requirements for functionality, integration, state, orchestration, security, deployment, and maintenance.

Before developing a custom agent: Consider whether an existing agent or visual builder already meets your needs. See Ready-to-Use AI Agent Comparison and Visual and Low-Code Agent Builder Comparison.

For the complete selection process and central weighted scorecard, see ...build an agent.

Product information reviewed: October 29, 2026. Framework versions, model compatibility, service availability, licensing, pricing, and documentation may change.

Quick Navigation

Mandatory Privacy Gate

Before choosing a framework, configuring tracing, or connecting real data, apply the central privacy gate to the entire application architecture. Framework choice does not determine the privacy of its model, tools, storage, monitoring, or managed execution services.

Surface to review Specialist checks
Model providers and routing Inventory inference endpoints, gateways, fallback models, embeddings, rerankers, and research APIs. Enforce permitted destinations and minimum payloads for every route.
Retrieval and tenant boundaries Authorize source access before retrieval, limit returned fields, and enforce user or tenant separation in tools, indexes, caches, sessions, and generated outputs. Do not depend on the model to enforce access rights.
State and memory Define owners, access, retention, and deletion for history, checkpoints, persistent memory, vector stores, caches, backups, and stored artifacts. Ensure resumption does not restore access that has been revoked.
Tracing, evaluation, and support Inspect what spans, tool arguments, outputs, exceptions, and evaluation datasets contain and where they are exported. Configure redaction, access, retention, or disabling before using restricted data.
Delegation and tools Treat each subagent, MCP server, plugin, code runner, search service, and external tool as an explicit boundary. Pass only necessary context and carry authorization through handoffs.
Managed or application-hosted execution For managed services, verify the exact service features and contract scope. For application-hosted systems, verify runtime, model, network, storage, and operator controls; self-hosting only one component is insufficient.

Build the privacy boundary into the application and its infrastructure. Guardrails and prompt instructions may help detect problems, but cannot replace authorization, tenant isolation, restricted destinations, or verified storage policies.

Gate outcome: Use the central definitions of Pass, Fail, and Unverified. Only configurations that pass for the intended data enter the approved shortlist. Failed configurations must be changed and reassessed; unverified candidates may receive only separately permitted public or synthetic-data trials.

The following product tables describe functional fit, not privacy certification. All quickstarts and exercises are subject to this gate.

Choose a Development Framework

Eligibility first: Use this section only within the data and deployment boundary established by the mandatory privacy gate.

Begin with the intended application and the type of execution control it requires.

The following table identifies the six frameworks, their primary strengths, and the development considerations that most strongly distinguish them.

Framework Best starting use Programming approach Principal consideration Detailed profile
OpenAI Agents SDK Building tool-using agents with handoffs, sessions, guardrails, and tracing Python and TypeScript Convenient code-first agent execution while retaining responsibility for application hosting and policies Profile and quickstart
Claude Agent SDK Embedding Claude's agentic file, command, and tool execution into applications Python and TypeScript Tool permissions, execution isolation, process management, and deployment Profile and quickstart
Google ADK Developing agents, tools, and coordinated workflows with Google and supported alternative models Python, TypeScript, Go, Java, Kotlin, subject to implementation-specific support Model compatibility, language-specific features, and deployment configuration Profile and quickstart
LangGraph Stateful workflows requiring explicit routing, checkpoints, and execution control Python and JavaScript/TypeScript Fine-grained control balanced against greater architecture and state-management effort Profile and quickstart
CrewAI Specialized agents coordinated through crews and structured flows Primarily Python Role-based collaboration balanced against coordination overhead and model consumption Profile and quickstart
Qwen-Agent Qwen-based applications requiring tool calling, retrieval, and extensible agent components Primarily Python Model-service configuration, tool-call parsing, and hosting requirements Profile and quickstart

Practical starting guidance:

  • First code-first agent: Investigate OpenAI Agents SDK or Google ADK.
  • Claude-based agent execution: Investigate Claude Agent SDK.
  • Stateful workflows and explicit branching: Investigate LangGraph.
  • Specialist collaboration: Investigate CrewAI.
  • Qwen model ecosystem: Investigate Qwen-Agent.
  • Provider-managed execution: Review Managed Agent Platforms and Specialized APIs before selecting an application-hosted framework.

These are starting recommendations, not independently established performance rankings.

Selection procedure: Choose two candidates that meet your mandatory requirements, complete their official quickstarts, and build the same representative application in both.

Understanding Agent Development

Framework, SDK, Model API, or Managed Service?

The following development technologies serve different purposes.

Technology What it provides What the developer typically controls
Model API Access to an AI model and its supported generation, reasoning, or tool-calling capabilities Agent execution, workflow logic, state, tools, application integration, and deployment
Agent SDK Reusable components for agent execution, tools, and supported orchestration Application code, business rules, hosting, security policies, and operational configuration
Orchestration framework Workflow routing, state transitions, coordinated execution, and sometimes persistence Workflow design, model integrations, storage, and application infrastructure
Managed agent platform Provider-operated agent execution or supporting infrastructure Agent configuration, connected tools, authorization, business logic, and service governance
Specialized research API Search, retrieval, source-based synthesis, or research-oriented capabilities The surrounding application and any additional workflow or action logic
Visual agent builder Graphical configuration of agents, integrations, knowledge, and workflows Requirements, configuration, integration, testing, and operational oversight

These categories overlap. A single vendor may offer model APIs, SDKs, hosted agent execution, and graphical tools as separate products.

Important: An agent framework does not necessarily provide hosted execution, persistent storage, user authentication, application permissions, or production monitoring automatically.

Components of a Custom Agent Application

A typical application may contain:

Component Responsibility
Application interface Receives user requests and presents results
Agent runtime Coordinates model requests, tools, and task execution
AI model Interprets information, generates output, and may select tools
Tool layer Connects the agent to permitted functions, APIs, databases, and services
Instructions and policies Define objectives, constraints, and expected behavior
Workflow state Records inputs, progress, intermediate results, and pending actions
Memory Preserves selected information across interactions or tasks
Approval controls Enforce authorization before consequential actions
Observability Provides execution logs, tracing, diagnostics, and usage information
Infrastructure Hosts, scales, protects, and operates the application

Not every application needs a separate service for each component. A smaller implementation may combine several responsibilities in one process.

Context, Sessions, State, and Memory

These concepts are related but distinct.

  • Context — Information supplied to the model for an individual interaction.
  • Conversation history — A record of earlier messages or exchanges.
  • Session — A defined sequence of related interactions or execution activity.
  • Workflow state — Current task inputs, intermediate results, status, and pending actions.
  • Persistent state — State saved outside a transient process so it can survive an interruption or restart.
  • Memory — Selected information retained for use across interactions or tasks.
  • Checkpoint — A recoverable snapshot or recorded execution position.

Key question: If the application stops unexpectedly, what information survives and how does execution resume?

A framework's support for conversation history does not automatically provide durable workflow recovery.

Common Agent Orchestration Patterns

Pattern How it works Typical use
Single agent with tools One agent selects among authorized functions Research, support, database lookup
Routing A coordinator or rule selects an appropriate specialist Directing customer requests to different functions
Manager and specialists A manager delegates subtasks and assembles results Complex research or multi-domain analysis
Sequential workflow Operations execute in a defined order Extract, validate, summarize, approve
Conditional workflow State or previous results determine the next operation Request clarification when information is missing
Parallel execution Independent tasks proceed concurrently Research several information sources
Human-in-the-loop Execution pauses for review or approval Authorizing external changes
Durable execution A saved execution state supports continuation Long-running or interruption-prone processes

Design principle: Begin with a single agent and a small number of tools. Add specialist agents, complex graphs, or long-running execution only when the application's requirements justify the additional complexity.

Framework Profiles and Official Quickstarts

The following profiles describe each framework's architecture, distinguishing capabilities, appropriate applications, limitations, and first practical implementation.

Official quickstarts are included with their respective profiles rather than repeated in a separate documentation directory.

OpenAI Agents SDK

Privacy checks for this configuration: Review OpenAI API data controls for the exact endpoints and features. Record tracing and storage destinations separately from model requests; do not assume API retention settings cover all application or tool data.

Provider: OpenAI

Primary purpose: Code-first development of AI agents with tools, handoffs, guardrails, sessions, and tracing.

The OpenAI Agents SDK supplies reusable execution components for Python and TypeScript applications.

The developer defines agents and their available capabilities, while the SDK manages the supported agent execution loop.

Distinguishing capabilities:

  • Agent instructions and configuration.
  • Function tools and supported hosted tools.
  • Agents used as tools.
  • Handoffs between specialist agents.
  • Session and conversation-state options.
  • Input and output guardrails.
  • Human approval and interruption mechanisms.
  • MCP integrations.
  • Execution tracing.
  • Supported sandbox-oriented agent capabilities.
  • Model-provider integration options.

Good candidate when:

  • You want a relatively direct programming model.
  • Your application requires structured tool use.
  • You want specialist delegation without creating your own agent execution loop.
  • Observability and tracing are important.
  • You want the agent runtime to operate within your application.

Evaluate carefully:

  • Compatibility with the required model and tool features.
  • State and session persistence.
  • Application hosting and scaling.
  • API consumption and tool charges.
  • Data recorded in traces.
  • Approval and permission enforcement.
  • Behavior when an agent or external tool fails.

Important distinction: OpenAI Agents SDK is application-hosted. The separate OpenAI Agents API provides a managed agent execution option.

OpenAI Agents SDK: Getting Started

  1. Open the official Agents SDK quickstart.
  2. Choose Python or TypeScript.
  3. Create a small development project.
  4. Install the supported SDK package.
  5. Configure model authentication securely.
  6. Define one agent with a narrow objective.
  7. Run a simple request.
  8. Add one read-only function tool.
  9. Inspect the response and execution information.
  10. Test an invalid input or tool failure before enabling additional actions.

Suggested first application: An agent that looks up a reference record through an authorized function and returns a structured summary.

Official quickstart: OpenAI Agents SDK Quickstart

Additional resources:

Claude Agent SDK

Privacy checks for this configuration: Map files, command output, session storage, hooks, and MCP traffic for the deployed process. Check the chosen model access route and terms separately from application-hosted runtime controls.

Provider: Anthropic

Related PRIMO.ai article: Claude

Primary purpose: Incorporating Claude's agentic execution capabilities into custom applications.

The Claude Agent SDK exposes the agent runtime associated with Claude Code through programmable interfaces.

This differs from using a conventional Claude API client to make individual model requests.

Distinguishing capabilities:

  • Python and TypeScript interfaces.
  • Agent execution with supported file and command tools.
  • Custom tools.
  • MCP integration.
  • Sessions and supported continuation mechanisms.
  • Subagents.
  • Permission handling.
  • Hooks.
  • Structured output.
  • Cost and usage reporting.
  • Execution observability.
  • Deployment and hosting guidance.

Good candidate when:

  • The application requires Claude-based agent execution.
  • Filesystem and development-style tools are important.
  • Custom tool permissions need careful configuration.
  • You want an agent runtime integrated with your own software.
  • Your organization can operate the required execution environment.

Evaluate carefully:

  • Tool and filesystem access.
  • Operating-system isolation.
  • Session persistence.
  • Process recovery.
  • Deployment requirements.
  • Model usage and execution charges.
  • Approval handling.
  • Security of custom and third-party tools.

Important distinction:

  • Claude API client SDK — Accesses Claude models through ordinary API requests.
  • Claude Agent SDK — Provides an application-controlled agentic execution process.
  • Claude Managed Agents — Provides separately managed agent execution.

Claude Agent SDK: Getting Started

  1. Open the official Agent SDK quickstart.
  2. Choose Python or TypeScript.
  3. Create a dedicated development project.
  4. Install the required package and prerequisites.
  5. Configure authentication.
  6. Define a narrow task.
  7. Restrict the tools available to the agent.
  8. Run a sample request.
  9. Inspect tool operations and output.
  10. Add permission handling before enabling file or system modifications.

Suggested first application: An agent that examines a sample project directory, identifies missing documentation, and prepares recommendations without automatically editing files.

Official quickstart: Claude Agent SDK Quickstart

Additional resources:

Google ADK

Privacy checks for this configuration: Verify model endpoints, session and artifact services, evaluation exports, and cloud identity in the selected language implementation. Local development does not establish a local data boundary for deployed services.

Provider: Google

Related PRIMO.ai article: Gemini

Full name: Agent Development Kit

Primary purpose: Building custom tool-using agents and multi-agent workflows using a code-first development framework.

Google ADK supports several programming languages and offers components for tools, workflow agents, state, evaluation, and deployment.

Distinguishing capabilities:

  • Code-defined agents.
  • Function tools and external-service integration.
  • Multi-agent composition.
  • Sequential and parallel execution.
  • Workflow routing.
  • Session and state management.
  • Local development interfaces.
  • Evaluation capabilities.
  • Gemini and supported alternative model integrations.
  • Deployment through compatible application and cloud environments.

Good candidate when:

  • You work in the Google AI or Cloud ecosystem.
  • Your application requires multiple tools or agents.
  • You want an agent framework supporting several programming languages.
  • You need structured workflows.
  • You plan to evaluate deployment through supported Google infrastructure.

Evaluate carefully:

  • Differences between language implementations.
  • Supported model and tool features.
  • Cloud authentication.
  • State storage and persistence.
  • Deployment services and costs.
  • Organizational access controls.
  • Evaluation and monitoring.

Language support: Google documents ADK implementations for Python, TypeScript, Go, Java, and Kotlin. Individual features may differ across languages.

Google ADK: Getting Started

  1. Open the official ADK quickstart for your preferred language.
  2. Install the development tools and ADK package.
  3. Create a small agent project.
  4. Configure model authentication.
  5. Define one agent.
  6. Run the agent locally.
  7. Add one read-only function tool.
  8. Inspect its output and execution history.
  9. Test an unsuccessful lookup or other expected failure.

Suggested first application: An agent that retrieves a record through an authorized function, validates its response, and returns a concise answer.

Official quickstart: Google ADK Python Quickstart

Additional resources:

Deployment distinction: Google ADK is a development framework. Gemini Enterprise Agent Platform provides related managed infrastructure and cloud services; they are not interchangeable products.

LangGraph

Privacy checks for this configuration: Define access and deletion for checkpointers, persistent stores, cached state, and tracing integrations. Ensure resumed graphs recheck authorization and cannot recover access to records that a user is no longer permitted to read.

Provider: LangChain

Primary purpose: Building stateful and long-running agent workflows using explicit control over execution and state transitions.

LangGraph provides lower-level orchestration primitives that can combine deterministic application logic with AI-driven processing steps.

Developers can use a graph of nodes and edges or the supported functional workflow approach.

Distinguishing capabilities:

  • Graph-based execution modeling.
  • Functional workflow API.
  • State representation.
  • Conditional routing.
  • Checkpointing.
  • Persistent execution state.
  • Human-in-the-loop interruption.
  • Resumption and recovery.
  • Streaming.
  • Subgraphs and coordinated agents.
  • Integration with supported model and tool providers.
  • Developer-controlled execution logic.

Good candidate when:

  • The application has important workflow branching.
  • Execution state must persist.
  • Work must pause and resume.
  • Deterministic processing must be mixed with AI decisions.
  • Detailed control over operations is important.
  • Multiple agents need explicit coordination.

Evaluate carefully:

  • Workflow design complexity.
  • Required developer expertise.
  • Persistence and checkpointer configuration.
  • Model and tool integration.
  • Deployment and infrastructure.
  • Retry and recovery behavior.
  • Debugging complex graph transitions.
  • Long-term maintenance.

Important distinction: LangGraph is not the same as a higher-level LangChain agent interface.

LangChain's higher-level agents may be easier for a beginner who does not yet need direct control over graph execution.

LangGraph: Getting Started

  1. Open the LangGraph quickstart.
  2. Choose the Graph API or Functional API.
  3. Configure a supported model provider.
  4. Define a simple tool.
  5. Specify the workflow state.
  6. Implement an initial processing node.
  7. Define the permitted transitions.
  8. Run the workflow using sample input.
  9. Inspect the resulting state.
  10. Add checkpointing if persistence is required.

Suggested first application: A support-question workflow that classifies a request, retrieves an authorized record, and prepares a response requiring approval.

Official quickstart: LangGraph Quickstart

Additional resources:

Architecture note: LangGraph can coordinate agentic and non-agentic steps in the same workflow. The application developer remains responsible for its authorization and business rules.

CrewAI

Privacy checks for this configuration: Limit context passed among agents and inspect crew memory, knowledge stores, flow persistence, and monitoring integrations. A specialist should not receive the full source collection merely because another agent can access it.

Primary purpose: Building applications around specialized agents, collaborative crews, and structured flows.

CrewAI distinguishes among agents with defined responsibilities, crews coordinating related tasks, and flows controlling the wider application process.

Distinguishing capabilities:

  • Role-oriented agents.
  • Defined tasks and expected outputs.
  • Agent collaboration.
  • Crews for coordinated work.
  • Flows for execution structure and state.
  • Tool integrations.
  • Knowledge and memory components.
  • Model-provider configuration.
  • Human input and approvals.
  • Testing and tracing.
  • Checkpointing and operational integrations.

Good candidate when:

  • Specialized responsibilities are meaningful to the project.
  • You want a clear distinction between task execution and workflow coordination.
  • The application benefits from multiple agents working on related tasks.
  • Your team primarily develops in Python.
  • You need a structured process around agent collaboration.

Evaluate carefully:

  • Additional model consumption.
  • Task delegation overhead.
  • Duplicated or conflicting agent work.
  • Data exchanged between specialists.
  • Flow state and recovery.
  • Tool compatibility.
  • Operational complexity.
  • Debugging and maintenance.

Recommended design: Begin with a simple Flow or single agent. Introduce a crew when specialist collaboration improves the result enough to justify the added complexity.

CrewAI: Getting Started

  1. Open the CrewAI quickstart.
  2. Review the Python and package prerequisites.
  3. Create a development project.
  4. Configure the model provider.
  5. Define one focused agent.
  6. Define a task and expected output.
  7. Run the example.
  8. Examine the result and execution information.
  9. Add a Flow when state or defined execution steps are required.
  10. Add a second specialist only when it addresses a demonstrated need.

Suggested first application: A research agent that gathers information and produces a short report. Add a reviewer only after measuring the single-agent result.

Official quickstart: CrewAI Quickstart

Additional resources:

Qwen-Agent

Privacy checks for this configuration: Verify hosted or self-deployed inference endpoints, document and embedding services, code-interpreter storage, and MCP tools. A local model does not make external retrieval or tool execution local.

Provider: Alibaba

Related PRIMO.ai article: Alibaba Qwen

Primary purpose: Building Qwen-based agents with open-source components for tool use, retrieval, and application extension.

Qwen-Agent supports compatible hosted Qwen services and supported self-deployed Qwen model environments.

Distinguishing capabilities:

  • Agent abstractions.
  • Qwen model integration.
  • Custom function tools.
  • Tool-call processing and parsing.
  • Multi-step tool use.
  • Retrieval-augmented generation.
  • Document processing.
  • Code interpretation.
  • MCP integrations.
  • Streaming responses.
  • Optional graphical interfaces.

Good candidate when:

  • Your application will use Qwen models.
  • You prefer an open-source framework.
  • You need custom tools.
  • Document retrieval is important.
  • You want to evaluate hosted and self-deployed model options.
  • Your development environment is Python-oriented.

Evaluate carefully:

  • Supported Qwen models and endpoints.
  • Hosted versus local inference.
  • Tool-call parser behavior.
  • Retrieval dependencies.
  • GPU and hosting requirements.
  • Security of code-execution tools.
  • Model and dependency licensing.
  • Maintenance and updates.

Model distinction: Qwen-Agent is a framework; Qwen models provide inference. The application must configure a compatible model service.

Qwen-Agent: Getting Started

  1. Open the official Qwen-Agent quickstart.
  2. Install its Python package and required dependencies.
  3. Choose a compatible hosted or self-deployed Qwen model.
  4. Configure authentication or the local model endpoint.
  5. Define a simple Assistant agent.
  6. Register one read-only function tool.
  7. Execute a test request.
  8. Inspect tool-call handling and the response.
  9. Verify the expected behavior with an invalid or unavailable data lookup.

Suggested first application: A question-answering agent that searches a controlled reference collection and calls an authorized record-lookup function.

Official quickstart: Qwen-Agent Quickstart

Additional resources:

Operational note: Model tool calling may depend on server-side parser configuration. Verify the selected deployment's tool behavior before enabling consequential actions.

Managed Agent Platforms and Specialized APIs

A custom application does not always need an application-hosted agent framework.

Some projects benefit from provider-managed execution. Others require a specialized research API rather than a complete agent runtime.

These alternatives should be evaluated according to the specific service they provide.

Managed Agent Platforms

Service Primary purpose Important consideration Official starting point
OpenAI Agents API Managed agent execution, sessions, orchestration, and supported sandbox tools Managed-service usage, execution policies, session behavior, and application integration Agents API Quickstart
Claude Managed Agents Anthropic-managed agent execution with supported sessions, tools, and sandbox environments Eligibility, beta limitations, data handling, and managed execution policies Managed Agents Quickstart
Gemini Enterprise Agent Platform Google Cloud model and agent infrastructure Cloud identity, available models, infrastructure, deployment, and service billing Getting Started
Microsoft Copilot Studio Managed enterprise agent building with business knowledge, connectors, and governance Tenant licensing, agent harness, Microsoft identity, permissions, and publishing Copilot Studio Quickstart
Baidu Qianfan Baidu enterprise agent and application services Regional support, product selection, language, data services, and commercial terms Qianfan Documentation

Important distinctions:

  • OpenAI Agents SDK runs in the developer's application; OpenAI Agents API provides managed agent execution.
  • Claude Agent SDK operates through a developer-controlled process; Claude Managed Agents offers a managed service.
  • Google ADK provides application-development components; Gemini Enterprise Agent Platform supplies related cloud and managed infrastructure.
  • Copilot Studio and Baidu Qianfan include visual or low-code services as well as developer-oriented options. Their features and hosting arrangements must be checked for the specific product selected.

Managed does not mean maintenance-free. Developers remain responsible for permissions, application integration, validation, cost management, and operational oversight.

Availability caution: Managed-agent products may have account, region, data-processing, preview, or compliance restrictions. Confirm current contractual eligibility before sensitive or production deployments.

Choosing Between an SDK and a Managed Service

Decision factor Application-hosted framework or SDK Managed agent service
Execution infrastructure Operated by the developer or organization Operated by the provider, subject to supported options
Application control Usually extensive over runtime integration Determined by managed-service capabilities
State and recovery Requires framework and storage configuration May include managed session or persistence capabilities
Security responsibilities Application permissions plus hosting and runtime security Application permissions plus service and provider configuration
Operations Deployment, scaling, monitoring, and maintenance Provider-managed components plus application monitoring and administration
Costs Model usage, hosting, development, and operations Managed execution, model usage, tools, and related service charges
Portability Depends on framework, models, and application design Depends on provider interfaces, sessions, and service features

Selection principle: Use managed execution when it meaningfully reduces the burden of operating the application without sacrificing essential functionality, security, or control.

Use an application-hosted framework when your deployment, tool integration, or execution requirements demand greater direct control.

Specialized Research APIs

Research APIs may be sufficient when the application primarily needs grounded information retrieval and synthesis.

Service Primary capability Suitable application Official resource
Perplexity API Platform Supported search and agentic research capabilities with web access, source-based responses, and related APIs Research-oriented applications requiring evidence from current information Perplexity API Platform
You.com Research API Multi-step research, information synthesis, and cited results Applications requiring documented research and source attribution You.com Research API

A research API may become one tool within a larger agentic application. It does not automatically provide the application's authentication, business logic, action permissions, or workflow state.

Recommendation: If the requirement is only to retrieve and synthesize information, begin with a research API before designing a complex autonomous system.

One Common Agent Development Exercise

Test-data boundary: Begin with public or synthetic information and test accounts containing no unrelated sensitive material. Passing these exercises does not by itself authorize production data; confirm the privacy gate for the production configuration before rollout.

A meaningful framework comparison requires the same application objective, test data, and success criteria across candidate environments.

Rather than comparing unrelated sample projects, use the following common exercise.

Project: Research and Approval Agent

Objective: Create a small agentic application that answers questions from a controlled reference collection and requests human approval before any simulated external action.

This exercise is suitable for single-agent frameworks and can be extended to stateful or multi-agent systems where appropriate.

Required behavior:

  1. Accept a user question.
  2. Retrieve information from an authorized source.
  3. Identify relevant supporting material.
  4. Produce a concise, structured answer.
  5. Identify the source supporting the answer.
  6. Record the task's current status.
  7. Prepare a simulated publishing or notification action.
  8. Stop and request approval before executing that action.
  9. Handle missing information, failed tools, and interrupted execution appropriately.

Use the same reference collection and example questions for every framework.

Stage 1: Build the Smallest Working Agent

Implement:

  • One agent.
  • One clearly defined objective.
  • One read-only lookup tool.
  • Structured input and output.
  • Basic error reporting.

Success criteria:

  • The tool is selected appropriately.
  • The retrieved information is handled correctly.
  • The answer matches the source material.
  • The output uses the required structure.
  • Failed lookups are reported without fabricated results.

Measure: Installation effort, amount of code, clarity of documentation, tool behavior, execution time, and model consumption.

Stage 2: Add Workflow State

Extend the application to record:

  • Original question.
  • Retrieved information.
  • Completed operations.
  • Current status.
  • Proposed external action.
  • Approval status.
  • Final outcome.

Success criteria:

  • The application can inspect its state.
  • Intermediate results remain available as required.
  • A task can identify whether it is awaiting approval.
  • Persistent state survives an application restart if that capability is required.

Important: Test persistence in the actual selected storage configuration, not just within one uninterrupted process.

Stage 3: Enforce Human Approval

Introduce a simulated publishing or notification tool.

Required behavior:

  1. Prepare a proposed action.
  2. Present its intended effects.
  3. Pause or block execution.
  4. Accept approval or rejection.
  5. Execute only when approval is granted.
  6. Record what happened.
  7. Avoid duplicate execution if approval or a request is repeated.

Success criteria:

  • Unapproved actions do not execute.
  • Rejected actions remain blocked.
  • Approved actions are performed only within the authorized scope.
  • The final state records the decision.
  • Duplicate actions are prevented or handled according to explicit rules.

An instruction telling the agent to request approval is not sufficient by itself. The surrounding application should enforce the approval boundary.

Stage 4: Test Errors and Recovery

Test the following conditions using nonsensitive sample data.

Test condition Expected behavior
Missing information Report the gap rather than inventing an answer
Unavailable source Report the failed lookup and preserve relevant context
Invalid tool response Handle or reject malformed information
Repeated request Avoid unintended duplicate external actions
Execution interruption Recover or resume according to the implemented architecture
Unapproved action Stop before the external operation
Untrusted source instructions Treat unrelated embedded instructions as source data rather than authoritative commands
Usage or time limit Stop or report the limitation without leaving an uncontrolled execution

Success criteria: The application responds predictably, preserves essential information, and provides enough diagnostic evidence to understand the failure.

Optional Stage 5: Introduce Specialist Agents

Use this extension only when multi-agent coordination is relevant to the actual application.

Create:

  • A research agent responsible for retrieving and organizing evidence.
  • A reviewer agent responsible for checking completeness and unsupported claims.

Compare this implementation against the original single-agent system.

Measure:

  • Answer accuracy.
  • Quality of supporting evidence.
  • Additional tool and model calls.
  • Coordination complexity.
  • Duplicate or conflicting work.
  • Execution time.
  • Human intervention.
  • Overall operating cost.

Decision principle: Retain the multi-agent architecture only if the benefit justifies the additional complexity.

Privacy Boundary Tests

Test Expected evidence
Tenant and tool isolation Create two synthetic users with separate records. Verify retrieval, tools, sessions, checkpoints, and caches enforce their separation, including guessed record identifiers.
Routing and delegation Use synthetic private fields and a controlled denied endpoint. Verify minimization and blocked transfers during normal execution, fallback, retries, specialist handoffs, and tool failure.
State and observability lifecycle Inspect traces, logs, memory, indexes, and persisted state. Exercise deletion and restart behavior, and verify revoked access stays revoked. Record backup and provider-retention limits from documentation.

Run applicable tests in a controlled environment with synthetic markers, never live secrets or private records. Inspect configuration and available access or network records as well as outputs. Missing visibility is an evidence gap, not proof that no transfer occurred. Where provider-side behavior cannot be observed, retain applicable documentation and contractual evidence.

Record Comparable Results

Evaluation dimension Evidence to record
Privacy architecture Gate status, tenant isolation, model and tool destinations, state lifecycle, trace handling, and delegation tests
Task completion Whether the application performs all required steps
Development effort Installation, configuration, coding, and troubleshooting
Tool integration Correct tool use and handling of failed calls
State and persistence Recording, inspection, recovery, and resumption
Human approval Enforcement of authorization boundaries
Reliability Behavior with missing information and unexpected failures
Observability Quality of logs, traces, and diagnostics
Security Credential handling, permissions, and execution isolation
Model compatibility Availability and behavior of required model features
Cost Model usage, service charges, hosting, and maintenance effort

Comparison procedure:

  1. Build the exercise with the first candidate.
  2. Run all relevant test cases.
  3. Record the observed results.
  4. Build the same application with the second candidate.
  5. Repeat the same tests and success criteria.
  6. Compare the evidence.
  7. Use the central Platform Comparison Scorecard for the final weighted evaluation.

Do not assign a final rating to a capability that has not been adequately tested.

Architecture, Security, and Operations

A working demonstration does not establish that an agent is ready for production.

Production applications require reliable identity, authorization, state, error handling, monitoring, cost controls, and operational maintenance.

Application Architecture Checklist

Before deployment, determine:

  • How users authenticate.
  • How the application verifies permissions.
  • Which model providers are permitted.
  • Which tools and data sources can be accessed.
  • Where credentials are stored.
  • Where execution state is persisted.
  • How approvals are enforced.
  • How timeouts and retries work.
  • How duplicate operations are prevented.
  • How execution can be cancelled.
  • What information is logged.
  • Who is responsible for maintenance and recovery.

A framework may supply some of these components, but the application architecture must establish how the complete system operates.

Tool Design and Security

Tools should have:

  • A narrow and documented purpose.
  • Clearly defined inputs.
  • Input validation.
  • Predictable outputs.
  • Meaningful error messages.
  • Least-privilege permissions.
  • Appropriate time and resource limits.
  • Logging for consequential actions.
  • Reliable handling of failure.

Prefer separate read-only and write operations where practical.

For example, permission to retrieve a customer record should not automatically grant authority to modify or delete it.

Recommended Approval Boundaries

Action Suggested initial control
Read approved reference information Allow within the user's authorized scope
Produce summaries or drafts Allow with review of significant claims
Modify shared records Require approval
Send external communications Require approval
Delete information Require explicit approval
Execute privileged commands Require strong isolation and explicit authorization
Make financial commitments Require human authorization
Change access permissions Retain human control

These are recommended starting policies. Verify how the chosen platform or application can enforce each required control.

Security principle: Model instructions alone cannot replace application-level authorization.

Reliability and Recovery

Operational safeguards may include:

  • Execution timeouts.
  • Bounded retries.
  • Rate-limit handling.
  • Checkpoints.
  • Persistent state.
  • Cancellation.
  • Idempotency controls.
  • Error classification.
  • Compensating actions for completed external operations.
  • Monitoring and alerts.

Some safeguards must be implemented in the surrounding application rather than in the agent framework.

Important: A retry after a partial failure can unintentionally repeat an external action. Design tools and state transitions to prevent or manage duplicate effects.

Observability

Maintain sufficient information to answer:

  1. What task was requested?
  2. Which model and configuration were used?
  3. Which tools were selected?
  4. What information was retrieved?
  5. Which operations succeeded or failed?
  6. Was approval required and granted?
  7. What output was produced?
  8. How much time and usage were consumed?
  9. Did the workflow complete?

Do not record sensitive information in logs or traces unnecessarily.

For related principles, see AI Trust & Evaluation and AI Experiments & Benchmarks Navigator.

Cost, Licensing, and Portability

The cost of an agent framework is not limited to its installation or package license.

An open-source framework may require paid model access, infrastructure, development, and maintenance.

Total Cost of Ownership

Cost category Examples
Framework or managed service License, subscription, or hosted execution fees
Model usage Input, output, and other billable inference operations
Tool usage Search, retrieval, external APIs, code execution, and related services
Infrastructure Hosting, databases, storage, containers, and networking
Observability Logging, tracing, monitoring, and evaluation
Development Architecture, implementation, integration, and testing
Maintenance Upgrades, troubleshooting, backups, and support
Human review Approval, correction, and operational oversight

Total cost = Service fees + Model usage + Tool usage + Infrastructure + Development and maintenance + Human review

Do not count charges twice when a service bundles several components into one fee.

Estimate Cost per Successful Task

For each candidate:

  1. Record the framework version and model configuration.
  2. Execute the common application exercise.
  3. Record successful and unsuccessful runs.
  4. Measure model and tool usage.
  5. Record execution time.
  6. Estimate infrastructure consumption.
  7. Include the effort required for human intervention.
  8. Calculate approximate cost per successful task.
  9. Project expected monthly usage.

Complex multi-agent applications may incur additional model calls and coordination costs without improving the result.

Cost principle: A framework with less code is not necessarily cheaper to operate.

Licensing and Ownership

Before adopting a framework or managed service, review:

  • Current repository or service licensing.
  • Commercial-use conditions.
  • Third-party dependencies.
  • Model-provider terms.
  • Redistribution obligations.
  • Data-processing requirements.
  • Support arrangements.
  • Version compatibility and update policies.

Framework licensing and model licensing are separate matters.

Long-Term Portability

Determine whether:

  • The application can use an alternative supported model.
  • Business rules remain independent of the agent framework.
  • Tools can be adapted to other environments.
  • State and data can be exported.
  • The application can move to another hosting service.
  • Agent instructions and tool definitions are under source control.
  • Another developer can maintain the application.
  • Proprietary managed services are essential to execution.

Design recommendation: Keep domain logic, authorization, validation, and essential application data separate from vendor-specific agent implementation where practical.

Make Your Final Selection

Choose the development environment that can reliably implement the required agentic functionality within acceptable technical, financial, operational, and security constraints.

The most popular or feature-rich framework is not necessarily the best fit.

Verify Mandatory Requirements

Reconfirm the privacy gate for the configuration being selected. A Fail or Unverified result excludes it from the approved shortlist for the intended data. Weighted scores apply only after mandatory requirements pass.

Also verify the remaining essential conditions below.

Examples include:

  • Required programming language.
  • Supported models or providers.
  • Necessary tool and data integrations.
  • Deployment and hosting constraints.
  • Information security requirements and a current privacy-gate Pass for the entire application architecture.
  • Required approval and audit controls.
  • Persistent state and recovery where necessary.
  • Production availability and support.
  • Budget and maintenance constraints.

A candidate that fails a mandatory requirement should not be selected based solely on its performance in less important areas.

Use the Central Platform Comparison Scorecard

The PRIMO.ai Platform Comparison Scorecard is the authoritative weighted evaluation method for the five-page selection system.

For agent development frameworks, pay particular attention to:

  • Task functionality.
  • Development and maintenance effort.
  • Tool and data integration.
  • State and orchestration.
  • Security and human approval.
  • Reliability and observability.
  • Total operating costs.
  • Deployment and portability.

Use the evidence recorded during the common development exercise.

Record Not Tested for any criterion lacking sufficient evidence.

The detailed scoring formula and weights are maintained in the master selector rather than duplicated in this article.

Document Your Decision

Retain the privacy-gate record with the decision: data classification, exact account and configuration, permitted providers and locations, retention and deletion settings, supporting evidence and dates, required reviewer, and change triggers. Keep this record private; do not publish sensitive evaluation details on this wiki.

Decision item Record
Application objective Required functionality and intended users
Selected framework Product, version, and programming language
Model configuration Provider, model, and service settings
Execution approach Application-hosted framework or managed service
Tool integrations Required APIs, databases, files, and external systems
State management Sessions, persistence, memory, and recovery configuration
Security controls Authentication, authorization, approval, and logging
Prototype results Evidence from the common application exercise
Cost estimate Development and ongoing operating expenses
Alternatives Other candidates evaluated
Limitations Missing features, workarounds, and risks
Maintenance responsibility Person or team responsible for the application
Re-evaluation date Date to review the decision

Final selection principle: Choose the simplest framework or managed environment that meets the application's mandatory requirements, produces reliable results, and remains practical to secure, maintain, and operate.

Related PRIMO.ai Resources

Retrieved from "https://primo.ai/w/index.php?title=Agent_Development_Framework_Comparison&oldid=51524"