Introduction: One platform, two ways to build agents

AI agents are moving beyond simple question-and-answer experiences. They can search information, call APIs, analyze files, coordinate multiple steps, and interact with business applications. As these solutions become more complex, developers need to decide not only which model to use, but also how the agent itself should be built and operated.

Microsoft Foundry Agent Service provides two main ways to create agents: prompt agents and hosted agents.

At first glance, the difference may seem straightforward. A prompt agent is configured using a model, instructions, and tools, while a hosted agent is built using code and deployed as a container. But the architectural difference goes deeper than that. It affects who owns the orchestration logic, how the agent is deployed, which protocols it can expose, how state is managed, how much control developers have, and what they need to maintain.

A prompt agent is often the right choice when the agent’s behavior can be expressed through instructions, model selection, and supported tools. A hosted agent becomes more useful when the application needs custom code, a particular agent framework, middleware, custom protocols, or control over the runtime.

This article explains both approaches, how they work, where they differ, and how to choose the right one for a real application.

What is Microsoft Foundry Agent Service?

Microsoft Foundry Agent Service is a managed platform for building, deploying, and scaling AI agents. It provides a common place to work with supported models, agent runtimes, tools, identities, and observability.

Foundry gives developers different levels of control. They can define an agent through configuration and let the platform run it, or write their own agent application and deploy that code as a hosted agent.

The platform also provides shared capabilities such as:

• Models: Access supported models from the Foundry model catalog.
• Tools and toolboxes: Connect agents to capabilities such as web search, file search, code interpreter, MCP servers, and other supported integrations.
• Identity and security: Use Microsoft Entra identities, role-based access control, and supported network-isolation options.
• Observability: Inspect agent execution through tracing and supported Application Insights integrations.
• Versioning and endpoints: Manage agent versions and invoke agents through managed endpoints.

The important point is that Foundry does not force every agent into one development model. It lets teams choose how much of the agent implementation they want to own and how much of the runtime they want the platform to manage.

Official overview: Agents in Microsoft Foundry.

What is a prompt-based agent?

A prompt-based agent, referred to in the current Microsoft documentation as a prompt agent, is an agent defined through configuration rather than a custom application container.

You configure the model, instructions, and supported tools. Foundry operates the runtime that coordinates the model and those tools. You do not need to write and operate the agent’s orchestration server, build a container, or manage its compute resources.

A prompt-agent definition generally brings together the following elements:

Component Purpose

Model Provides the language and reasoning capabilities

Instructions Define the agent’s role, behavior, boundaries, and expected output

Tools Allow the agent to retrieve information or perform operations

Skills Provide reusable instructions, scripts, and resources for specialized tasks

Toolboxes Group reusable tools behind a managed MCP-compatible endpoint

Connections and permissions Control access to tools and downstream resources

For example, imagine a research assistant that needs to answer questions using a collection of technical documents. You can configure its instructions, select a supported model, attach file search or Azure AI Search, and test its behavior in the Foundry portal.

When a user asks a question, Foundry runs the agent using its saved definition. The agent can make model and tool calls as needed, then return the response.

How does a prompt agent work?

A saved prompt agent is a versioned resource in a Foundry project. When an application invokes it, the flow is broadly as follows:

1. The application sends the user’s input to the prompt agent’s Responses endpoint.
2. Foundry loads the saved agent definition, including its model, instructions, and configured tools.
3. The managed runtime coordinates model calls and supported tool calls.
4. The runtime returns the response.
5. Foundry records the request traces and, when a conversation is used, appends the response items to that conversation.

The developer defines the agent’s behavior and supported capabilities, while Foundry operates the runtime.

How can prompt agents be created?

There are several supported authoring paths:

Foundry portal: Create the agent interactively, configure its model and tools, and test it in the playground. This is useful when exploring an idea or validating instructions before integrating the agent into an application.

SDK or REST API: Define and manage the agent programmatically. This is useful when agent definitions need to be reviewed, version-controlled, and deployed through an engineering workflow.

Azure Developer CLI (azd): Keep the agent definition in the project and use the Azure Developer CLI to support repeatable deployment workflows.

A prompt agent is not limited to prototypes. Microsoft positions prompt agents for quick starts, internal applications, and production scenarios that do not require custom orchestration logic.

Official documentation: What are prompt agents?.

What is a hosted agent?

A hosted agent is a code-based agent application that you build and deploy to Microsoft Foundry Agent Service.

Instead of defining all behavior through an agent configuration, you write the application logic yourself. You can use a supported framework such as Microsoft Agent Framework, LangGraph, the OpenAI Agents SDK, the Anthropic Agent SDK, or your own code, subject to the current platform support requirements.

You package the application as a container image, or use the supported source-code ZIP deployment path where Foundry builds and hosts the image. Foundry then runs the agent on Microsoft-managed infrastructure and provides a managed endpoint.

The key distinction is:

You own the agent code and orchestration logic; Foundry manages the hosting platform around it.

This gives developers more control over how the agent behaves. They can implement custom routing, middleware, business logic, state handling, framework-specific workflows, and protocols that are not naturally expressed through a prompt-agent definition.

How does a hosted agent work?

The deployment and runtime flow looks like this:

1. Build the agent application using a supported framework or custom code.
2. Package the application as a container image and push it to Azure Container Registry, or use the supported ZIP deployment option.
3. Create an agent version in Foundry.
4. Foundry provisions the hosted runtime and creates a dedicated Microsoft Entra agent identity.
5. The platform exposes the agent’s endpoint.
6. The application sends requests to that endpoint, and the hosted agent code handles the request.

At runtime, the agent code can call Foundry models through the project endpoint and use supported Foundry tools and toolboxes. It can also interact with other services, provided the required connectivity and permissions are configured.

Foundry manages important hosting responsibilities such as endpoint provisioning, scaling, session-state persistence, and platform observability. Developers remain responsible for their application code, dependencies, workflow logic, testing, and application-level security.

What makes hosted agents different?

Hosted agents are useful when an application needs capabilities such as:

• Custom orchestration logic and framework-specific execution.
• Custom middleware, libraries, and runtime dependencies.
• Multi-agent coordination implemented in code.
• Custom HTTP, webhook, or streaming protocols.
• Stateful workloads that use the hosted session filesystem.
• More direct control over CPU and memory allocation.
• Long-running workflows that need application-level recovery behavior.

Official documentation: What are hosted agents?.

Prompt-based vs hosted agents: The core differences

Both agent types can use Foundry models and supported platform tools. Both can be integrated into applications, versioned, and observed. The main difference is where the agent’s orchestration logic lives and how much runtime control the developer needs.

Area Prompt agent Hosted agent

Agent definition Model, instructions, Application code and tools, and supported framework configuration

Main authoring surface Foundry portal, SDK, Agent framework or REST API, or azd custom code

Orchestration Managed by Foundry’s Implemented in your selected prompt-agent agent application runtime

Runtime code to No custom orchestration Yes, the agent maintain server required application must be maintained

Container deployment Not required Container or supported ZIP-based deployment

Compute management Fully managed by Container compute is Foundry hosted and scaled by Foundry

Framework choice Uses the supported Can use supported prompt-agent runtime frameworks or custom code

Custom middleware Not the primary Can be implemented in extension model application code

Custom protocols Limited to the Supports Responses, prompt-agent interface Invocations, and and supported WebSocket invocation capabilities protocols, among others where configured

Conversation handling Managed through the Depends on the prompt-agent runtime protocol; Responses and conversation supports features platform-managed conversation history

Session filesystem Not the same Persistent $HOME and per-session /files session hosted-container storage filesystem model

Identity Foundry-managed agent Dedicated Entra agent identity identity created for each deployed hosted agent

Cost considerations Model inference and Model inference, tool tool usage usage, and hosted container compute

Best fit Agents whose behavior Agents requiring custom can be defined through code, runtime behavior, configuration or protocols

The table describes the general architecture. Exact tool availability, framework support, and regional capabilities should be checked against the current Foundry documentation.

Understanding the architecture of both approaches

The easiest way to understand the difference is to look at how a request travels through each runtime.

Prompt-agent architecture

In a prompt-agent architecture, the application sends the request to the Foundry endpoint. Foundry loads the saved agent configuration and coordinates the model and configured tools.

The application does not need to run a separate orchestration server for the prompt agent. Foundry manages the execution of the saved definition.

Hosted-agent architecture

In a hosted-agent architecture, the request reaches the Foundry endpoint and is routed to the hosted agent’s runtime. The container runs the code that the developer has implemented.

The developer controls the application logic inside the hosted runtime, while Foundry provides the managed hosting environment, endpoint, identity, and lifecycle services.

Understanding hosted-agent protocols

One important advantage of hosted agents is that they can expose different protocols depending on the application requirement.

The protocol determines the request format, how the application communicates with the agent, and how much lifecycle handling is provided by the platform libraries.

Microsoft documents three main hosted-agent interaction protocols.

Responses protocol

The Responses protocol is designed for conversational agents.

It uses an OpenAI-compatible request and response contract. It supports multi-turn conversations, streaming, and platform-managed conversation history.

It is a good starting point for:

• Conversational assistants.
• Research agents.
• Multi-turn question-answering systems.
• Agents using retrieval-augmented generation.
• Agents that need tool execution and response streaming.

The protocol libraries handle important response lifecycle details, including streaming events, background execution, cancellation, and response management.

Invocations protocol

The Invocations protocol is useful when the caller needs to send a custom payload rather than a conversational message.

For example, a webhook may send structured JSON when a repository event occurs. A processing application might send a document identifier and processing parameters rather than a chat message.

With Invocations, the developer defines the request and response schema and manages session state in application code.

Typical scenarios include:

• Webhook receivers.
• Classification and extraction workloads.
• Batch processing.
• Custom asynchronous workflows.
• Integration with systems that use their own request formats.

Invocations over WebSocket

The WebSocket invocation protocol supports bidirectional streaming over a persistent connection.

This is useful for real-time interactions where the client continuously sends data and receives responses, such as audio streaming or interactive media applications.

Protocol Best suited for Main characteristic

Responses Chatbots and multi-turn Platform-managed agents conversation and streaming lifecycle

Invocations Webhooks and structured Developer-defined JSON processing request and response

Invocations WebSocket Real-time voice and Bidirectional interactive streaming communication

A hosted agent can expose multiple protocols in the same container when configured correctly. The choice is therefore not necessarily permanent.

For a conversational agent, Microsoft recommends starting with Responses. Invocations is more appropriate when the caller needs full control over the request contract.

Reference: Hosted agent protocols.

State management: Conversations vs sessions

State management is another important difference, especially when building agents that need to work across multiple interactions.

A prompt agent uses Foundry’s managed conversation capabilities. The application can pass a conversation identifier so that the runtime can continue the conversation using its stored history.

Hosted agents have a richer session model because the application runs inside a managed execution environment.

Three concepts are important:

Sessions: A session identifies a logical execution context. Hosted agents can persist files in $HOME and content uploaded through the /files endpoint across turns and idle periods.

Conversations: A conversation stores durable interaction history, including messages, tool calls, and responses. Under the Responses protocol, Foundry manages conversation history.

State store: Hosted agents can use a durable server-backed key-value store for application state that should survive container restarts and idle eviction. This is useful for framework checkpoints and other application-specific data.

How hosted-agent session persistence works

Hosted agents run in per-session VM-isolated sandboxes. Each session has its own execution environment and filesystem.

When a session becomes idle for its configured period, Foundry can deprovision its compute while preserving the supported session filesystem state. When the session is used again, the platform provisions compute and restores the saved state.

The documented idle timeout can be configured from 2 to 240 minutes, with a 15-minute default. A session is permanently deleted after 30 days of inactivity.

This does not mean every variable in application memory is automatically preserved. Developers should use the documented persistent filesystem or state store for information that must survive compute deprovisioning.

The protocol also affects state handling:

• With Responses, the conversation ID is the primary interaction concept, and Foundry manages conversation history.
• With Invocations, the session ID is the primary concept, and the application manages its own session state.

Reference: Hosted-agent sessions, conversations, and state store.

Deployment: What changes between prompt and hosted agents?

The deployment process is one of the clearest differences.

Deploying a prompt agent

A prompt agent can be created in the Foundry portal or defined through the SDK, REST API, or Azure Developer CLI.

The general process is:

1. Create a Foundry project and select a supported model.
2. Define the agent instructions.
3. Configure supported tools, skills, and toolboxes.
4. Test the agent in the playground.
5. Save the agent definition as a version.
6. Invoke the agent through its Responses endpoint.

There is no need to build an agent container or manage a separate orchestration server.

This makes the prompt-agent approach useful for teams that want to move quickly from an idea to a working agent.

Deploying a hosted agent

A hosted agent involves an application deployment workflow.

The general process is:

1. Create the agent application using a supported framework or custom code.
2. Implement the request handler and the required agent logic.
3. Configure the appropriate protocol library.
4. Build and package the application.
5. Push the container image to Azure Container Registry, or use the supported ZIP deployment workflow.
6. Create a hosted-agent version in Foundry.
7. Wait until the version reaches the active state.
8. Invoke the agent through its dedicated endpoint.

Microsoft also supports deployment through the Azure Developer CLI (azd), SDKs, and REST API. The exact requirements depend on the deployment path.

A hosted deployment has additional engineering requirements, such as container compatibility, dependencies, permissions, and protocol configuration. For example, the hosted-agent platform requires Linux x86_64 (linux/amd64) container images.

The deployment tooling can automate several steps, but developers still need to maintain and test the application code.

Reference: Deploy a hosted agent.

Identity, security, and access control

Both prompt and hosted agents integrate with Microsoft Entra identity and Foundry’s security capabilities, but hosted agents have an additional runtime identity model.

Every deployed hosted agent receives a dedicated Microsoft Entra agent identity. The container uses this identity when accessing models, tools, and downstream Azure resources.

The identity has default access to model inference through the Foundry project endpoint and session storage. If the agent needs to access an external resource, such as a separate Azure Storage account, the required RBAC permissions must be assigned.

For example, a hosted document-processing agent might need read access to a storage account. Developers should grant the agent identity only the permissions required for that task.

Prompt agents also use managed identity and connection-based authorization for their configured capabilities. The exact identity and permissions depend on the tool and connection.

In both approaches, developers must consider:

• Least-privilege access.
• Secure credential storage.
• Data access permissions.
• Tool input and output validation.
• Responsible AI controls.
• Network isolation requirements.
• Third-party data handling and compliance boundaries.

An important point from Microsoft’s hosted-agent guidance is that developers remain responsible for reviewing the data shared with third-party services and implementing appropriate application-level safety mitigations.

Hosted infrastructure does not automatically make custom application code safe or compliant.

Scaling, performance, and cost considerations

Both prompt and hosted agents benefit from managed scaling, but their compute models differ.

Prompt agents

Foundry manages the prompt-agent runtime and scales it according to service behavior and request volume. Developers do not manage a container or specify its CPU and memory allocation.

The documented cost model is based on model inference and tool usage.

This makes prompt agents attractive when the workload can be handled by the managed runtime and configured capabilities.

Hosted agents

Hosted agents use per-session VM-isolated sandboxes. Foundry provisions compute when a session is used and deprovisions it after the configured idle timeout, while preserving supported session state.

Developers can configure CPU and memory resources for the hosted runtime. The platform manages provisioning, lifecycle, and scaling, but the hosted application introduces a compute cost in addition to model inference and tool usage.

The practical cost depends on:

• Number of active sessions.
• CPU and memory configuration.
• Session activity and idle timeout.
• Model usage.
• Tool usage.
• External services called by the agent.

Hosted agents are not automatically more expensive in every workload, and prompt agents are not automatically cheaper for every application. The correct comparison requires estimating the actual request volume, model consumption, tool calls, and compute activity.

For current costs, check the Microsoft Foundry pricing page.

When should you choose a prompt agent?

Prompt agents are a good choice when the required behavior can be described through instructions, model selection, and supported tools.

Consider a web research assistant. It receives a question, searches for relevant information, retrieves supporting content, and summarizes the findings.

If the workflow can be handled through the configured tools and the managed prompt-agent runtime, there may be no reason to build and operate a separate containerized application.

Prompt agents are especially suitable when:

• You want to create an agent quickly.
• The agent’s behavior is primarily instruction-driven.
• Supported tools provide the required capabilities.
• You do not need custom orchestration code.
• You want Foundry to manage the runtime and compute.
• You need a saved, versioned agent resource.
• Your team prefers portal-first development or declarative configuration.

Prompt agents can also be used in production. The decision should be based on the required runtime capabilities, not on the assumption that prompt agents are only for demonstrations or prototypes.

When should you choose a hosted agent?

Hosted agents become useful when the application requires logic that goes beyond the prompt-agent configuration model.

Consider a developer assistant that needs to inspect a repository, decide which specialized agent should handle a task, execute custom validation logic, maintain framework checkpoints, and communicate with an external development system through a webhook.

This workflow may require a custom orchestration layer, additional dependencies, and application-specific state management.

A hosted agent gives developers the flexibility to implement those requirements in code while using Foundry for managed hosting.

Choose a hosted agent when:

• You need to use a particular supported agent framework.
• You have custom orchestration logic.
• You need custom middleware or runtime dependencies.
• You are building multi-agent workflows in code.
• You need to accept webhooks or custom JSON payloads.
• You need bidirectional WebSocket streaming.
• You need session-level filesystem persistence.
• You need to control the hosted compute resources.
• You need application-specific recovery or execution behavior.

The important point is that hosted agents provide more control over the application runtime, but they also introduce more engineering responsibility.

Practical scenarios: Which architecture fits?

The following examples illustrate how to make the choice.

Scenario Recommended starting Reason point

Document Prompt agent Instructions and file question-answering search or Azure AI agent Search may be sufficient

Web research assistant Prompt agent Search tools and instructions can handle the workflow

Structured data Prompt agent or hosted Prompt agent for extraction agent supported extraction; hosted when custom processing logic is needed

Custom multi-agent Hosted agent Application code orchestration controls delegation and workflow

Repository event Hosted agent Invocations supports webhook custom request payloads

Real-time audio Hosted agent or Depends on whether processing voice-based prompt custom voice agent orchestration is required

Custom framework-based Hosted agent Useful when retrieval RAG pipeline and orchestration need application code

Simple API-connected Prompt agent Supported tools and assistant configuration may be sufficient

Custom streaming Hosted agent More control over the interface protocol and streaming behavior

These are architectural starting points, not strict product restrictions. The right decision depends on the exact capabilities available in the selected region, model, runtime, and tool configuration.

Is there a third option? Using the Responses API directly

There is another approach worth understanding.

If your application already owns the agent definition and does not need a saved prompt-agent resource, you can call the Foundry Responses API directly.

In this approach, the application provides the model, instructions, and tools for each request. Foundry provides the model inference and supported platform capabilities, but you do not create a persistent prompt-agent resource.

This is sometimes described as an ephemeral agent pattern.

It can be useful when:

• The agent definition changes dynamically for each request.
• The application already manages its own orchestration.
• You want to use Foundry models and tools without maintaining a separate saved agent resource.
• You want the agent configuration to live and be versioned alongside the application code.

There are trade-offs. The ephemeral pattern does not support capabilities that require a saved prompt-agent resource, such as selecting the Copilot harness or using prompt-agent optimization.

It is also important not to confuse this pattern with a hosted agent. A direct Responses API integration does not mean your application has been deployed as a hosted agent.

Reference: Use the Responses API directly.

A note on the Copilot harness: Prompt agents can do more than basic tool calling

One feature worth mentioning is the Microsoft Copilot harness, which is documented as a preview option for prompt agents.

The default prompt-agent runtime coordinates the model and the tools declared in the agent definition. The optional Copilot harness provides a managed execution environment for supported multi-step tool, skill, file, and command workflows.

It can provide capabilities such as file-system operations, shell commands, web fetching, and subagent workflows, alongside compatible declared tools and skills.

This is important because developers sometimes assume that any workflow involving multiple steps or tools automatically requires a hosted agent.

That is not necessarily true. If the required capabilities are supported by the prompt agent and its selected harness, Foundry can manage that execution without the developer hosting the harness code.

However, the harness has its own preview limitations and security considerations. For example, developers should review the built-in tools carefully before enabling broad capabilities such as shell commands. Region, model, tool, and conversation requirements must also be checked.

The Copilot harness remains a prompt-agent runtime choice; it does not turn the resource into a hosted agent.

Reference: Choose a managed runtime for prompt agents.

A practical decision-making guide

Before selecting an agent type, answer a few architectural questions.

Can the workflow be expressed through instructions and supported tools?

If yes, start with a prompt agent. Avoid adding a custom runtime unless the application needs one.

Do you need to write your own orchestration code?

If the workflow requires custom routing, conditional execution, framework-specific checkpoints, or middleware, consider a hosted agent.

Does the application need a custom protocol?

For webhooks, custom JSON contracts, or bidirectional WebSocket interactions, hosted agents provide the appropriate protocol options.

Does the application need persistent execution state?

If you need a session filesystem, framework checkpoints, or custom state management, examine the hosted-agent session and state-store capabilities.

Do you want to manage container dependencies and application code?

If not, a prompt agent is generally simpler. If you need those dependencies and runtime controls, a hosted agent is more appropriate.

A useful rule is:

Start with a prompt agent when configuration is enough. Move to a hosted agent when the application needs code-level control.

This avoids introducing container deployment and application-runtime responsibilities before they are actually required.

Conclusion: Choosing the right balance between simplicity and control

Microsoft Foundry provides two distinct approaches to building AI agents.

Prompt agents make it possible to create and operate agents through model selection, instructions, and supported tools without maintaining a custom orchestration runtime. Hosted agents allow developers to bring their own code and frameworks while using Foundry’s managed infrastructure for deployment, identity, scaling, and observability.

Neither approach is universally better. The choice depends on the complexity of the workflow, the amount of custom logic required, the communication protocol, state-management needs, and the operational responsibilities the development team is prepared to own.

For many applications, starting with a prompt agent is the simplest path. When custom orchestration, framework integration, or runtime-level control becomes necessary, a hosted agent provides a more flexible architecture.

The goal is not to choose the most complex agent architecture. It is to choose the simplest architecture that can reliably support the application’s requirements.

References

1. Microsoft Learn — Agents in Microsoft Foundry