Hosted Agents in Microsoft Foundry - code over prompts
In June 2026, I stumbled upon an announcement that Hosted Agents functionality of Microsoft Foundry went live, and promised the ability to properly code the agents that are deployed on the Foundry instances.
As I already had some experience in the past with hosting different agents, I took that news with great joy. As an engineer, I am definitely more fond of writing code that solves the problem, instead of relying on the prompts alone (even if I mostly work starting from prompt-engineering nowadays).
Before Hosted Agents, the approach to use agents would require going one of two ways. One was to configure everything on the Microsoft Foundry instance - prompts, tools, data sources - and then sending the end user endpoint URL and access key that they could utilize for their needs. Second option was to just provide endpoint and the access key, expecting user to provide all the tooling (and e.g. credentials) during the call.
While I may be getting a little bit paranoid, option one was always more reasonable for me, as it would allow us to avoid having sensitive data transfer over internet connection. All the credentials and configuration were still living in the Microsoft Foundry - so if you wanted to set-up another instance of the agent, you would either have to click-ops through the portal again, or write some elaborate REST API calls/CLI scripts that would set it up for you.
This is where Hosted Agents bring us much closer to what as an engineer I am more familiar with - you deploy the whole set-up as a docker image, which loads environment variables for things that may be instance-specific. Finally, the business logic goes back where it belongs - to the source code. If I want to make my agent have access to the Storage Account, I can simply write proper code to include such functionality and include information in the instruction set, that such a command exists in case it would be necessary. It’s good to finally go back to relying on code as main source of truth.
Code-first approach
As I already mentioned, Hosted Agents rely on Docker containers. Microsoft documentation provides further split based on functionality of the Agent - into responses (typical LLM chatbot based on OpenAI schema) and invocations (much more customizable, but also more complex in implementation).
Responses by default support all standard OpenAI SDK platforms, while samples focus mostly on Python and C#. Then, it also distinguishes between bring-your-own code approach and relying on Microsoft Agent Framework to utilize some of the built-in functionalities that Microsoft Foundry provides.
Great news is that the number of samples provided on the GitHub is quite large and covers a lot of scenarios. While in theory this can be overwhelming for one person to learn, for coding agents that can just fetch those examples and build the solution based on it, it’s a great starting spot. Of course, at the end of the day it’s still your own responsibility to understand what the code does, but that’s nothing new.
As this relies on Docker images, you also need to host the Docker images somewhere - so registry, such as Azure Container Registry is another resource that is needed for such a set-up, that was not necessary for the stand-alone Microsoft Foundry agents. This includes some additional complexity, as you now need to be sure that Foundry can reach that container, but utilizing built-in things like managed identity makes it a rather trivial thing.
Tricky part is that you are not able to identify which ones were deployed via Hosted Agents vs ‘standard ones’ - you will still see the same list of agents as earlier, and the only source of observability is the App Insights you connect to the agent - if you did that in the first place. I will also come back to this topic later, when discussing how the troubleshooting for my first deployment went.
Let’s deploy a hosted agent
To run anything, first you need a proper infrastructure in place - in theory you are able to run that with just Microsoft Foundry and Azure Container Registry for storing the containers with your hosted agent codebase. You can select if you want to do it via Terraform, Bicep, ARM or click-ops via Portal.
For proper agent deployment, you have to use Azure Developer CLI (azd). Then, you need to create new environment or select an existing one via azd env, to set-up where the deployment should happen. Similarly, you have to point out to which project in Microsoft Foundry you are connecting via azd ai project set command.
Afterwards, you can deploy the agent via azd deploy, assuming you have the proper YAML file with definition of the agent. You can find the example one in my Hosted Agents lab repository here, and see how it’s being deployed via pipeline here.
What is interesting, for the deployment YAML the structure looks a little similar to the standard Kubernetes/Helm chart syntax. Which makes sense, taking into consideration that under the hood this uses Azure Container Apps, which share some similarities with typical Kubernetes deployment.
What does the code do?
In case of my GitHub repository, the goal was to create the end-to-end set-up on how to prepare code and deployment for Hosted Agents.
Codebase creates a single hosted agent, that has two OpenAI SDK-connected tools (prepared using Azure.AI.Extensions.OpenAI NuGet package) - Storage Account access and Key Vault access. Depending on the prompt, it will reach either one. Default examples available on Microsoft GitHub repository (which also offers the connection to Storage Account) expected that agent will connect each time, while this repo does it only on-demand. A little improvement over the basics. Also, both connections are done via Managed Identity, rather than passing the credentials in the environment variables.
Terraform code is pretty self explanatory - it creates Microsoft Foundry, internal project, Storage Account, Key Vault, Container Registry and App Insights for observability purposes - pretty simple stack, but something that can be easily copied for your own projects. While it was tempting to rely on Azure Verified Modules, I decided to go with simpler approach - AVMs are prepared for prod-ready deployments, while purpose of this lab is to mostly showcase how to work with that technology. Thanks to that you may get a clearer image how to incorporate into your existing code base.
Last part of code is the client - application that will connect to the Hosted Agent and can prompt it for answers to the questions. This is really a simple wrapper over the REST API, that is there mostly for demos during my potential meetups/talks.
Getting proper permissions in place
As part of the deployment setup, my agent was supposed to have new managed identity assigned. By design I am trying to avoid using user-assigned identities wherever possible, so I went with the default solution. And I wanted to have that agent utilize access to my key vault and storage account.
With System-assigned identity created as part of the deployment, I was unable to know the proper object ID before the agent was ready. Because of that, I had to assign the proper RBAC roles after the agent started running, which may be the issue when you want agents to utilize some sensitive values that may be stored e.g. in the key-vault. Please keep in mind, that in such cases you need to either provide the values to the agents via environment variables, or rely on managed identity but expect in code that RBAC role may be assigned post-deployment. And in case you would like to rotate some of those sensitive secrets, you are probably going to have to re-deploy the whole agent.
I also tried with the typical Azure set-up for passing the secrets via @Microsoft.KeyVault(...) type argument. It was being treated as literal value, rather than resolved at runtime. Please keep that in mind.
Troubleshooting the initial deployment
By default I am usually trying to aim for deploying everything for my PoCs in Poland Central. As I checked in the Hosted Agents Microsoft Learn page, it is one of the available regions. Which is great news - I was worried that I will get warned to avoid West Europe region, as in my last few attempts working with it, I faced the issue with AKS availability - and after all, Hosted Agents runs on Container Apps, which usually also relies on Kubernetes Cluster under the hood.
Once the deployment was completed, I started getting 404 “Project not found” error. Invoke calls returned 404 Project not found even though the project existed in Terraform and az. This error also popped up later on when I tried to connect to that agent without the required Foundry User RBAC role and received that error again - when I rather expected a 401 error instead.
Observability is definitely not the highlight of this set-up. I was unable to get debugging data from Microsoft Foundry - there is no dedicated diagnostic settings for hosted agents, you have to rely on what is being passed to App Insights. And if your hosted agent doesn’t work, it won’t push anything there. Then, you start guessing and hoping something will stick. In my case, nothing did.
Then I gave up, and re-used the exact same code for deployment in West Europe. Everything worked out of the box. So, in case you would try to use this functionality and for some reason you get unexpected errors, try different regions - being available clearly does not mean it works.
Summary
While it’s still pretty young technology, I am very happy that we are finally getting back into the direction of writing proper code that utilizes LLM for some of the behavior, rather than just relying on how prompts will interpret what we type into it. I see potential in where it is going and will probably prepare some more PoCs based on that - hopefully in the meantime the solution will get stable enough to be properly used for something more production-ready or enterprise grade.
In case you are interested in how this works, I would highly recommend looking over my GitHub repository, where I prepared some materials:
- code for example agents that utilize Storage Account and Key Vault access
- client for connecting to that agent
- Terraform code for deploying the necessary infrastructure
- GitHub Action workflow for connecting it all together and deploying on Azure
Maybe you will find it useful, maybe your AI agent that will scrape it for your project and you’ll either way benefit from it. Enjoy!