Install and run Azure Foundry Local LLM server & Open WebUI on Windows Server 2025

Foundry Local is an on-device AI inference solution offering performance, privacy, customization, and cost advantages. It integrates seamlessly into your existing workflows and applications through an intuitive CLI, SDK, and REST API. Foundry Local has the following benefits:

• On-Device Inference: Run models locally on your own hardware, reducing your costs while keeping all your data on your device.
• Model Customization: Select from preset models or use your own to meet specific requirements and use cases.
• Cost Efficiency: Eliminate recurring cloud service costs by using your existing hardware, making AI more accessible.
• Seamless Integration: Connect with your applications through an SDK, API endpoints, or the CLI, with easy scaling to Azure AI Foundry as your needs grow.

Foundry Local is ideal for scenarios where:

• You want to keep sensitive data on your device.
• You need to operate in environments with limited or no internet connectivity.
• You want to reduce cloud inference costs.
• You need low-latency AI responses for real-time applications.
• You want to experiment with AI models before deploying to a cloud environment.

You can install Foundry Local by running the following command:

winget install Microsoft.FoundryLocal

Once Foundry Local is installed, you download and interact with a model from the command line by using a command like:

foundry model run phi-4

This will download the phi-4 model and provide a text based chat interface. If you want to interact with Foundry Local through a web chat interface, you can use the open source Open WebUI project. You can install Open WebUI on Windows Server by performing the following steps:

Download OpenWebUIInstaller.exe from https://github.com/BrainDriveAI/OpenWebUI_CondaInstaller/releases. You'll get warning messages from Windows Defender SmartScreen. Copy OpenWebUIInstaller.exe into C:\Temp.

In an elevated command prompt, run the following commands

winget install -e --id Anaconda.Miniconda3 --scope machine

$env:Path = 'C:\ProgramData\miniconda3;' + $env:Path
$env:Path = 'C:\ProgramData\miniconda3\Scripts;' + $env:Path
$env:Path = 'C:\ProgramData\miniconda3\Library\bin;' + $env:Path

conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
conda.exe tos accept --override-channels --channel https://repo.anaconda.com/pkgs/msys2

C:\Temp\OpenWebUIInstaller.exe

Then from the dialog choose to install and run Open WebUI. You then need to take several extra steps to configure Open WebUI to connect to the Foundry Local endpoint.

1. Enable Direct Connections in Open WebUI
   1. Select Settings and Admin Settings in the profile menu.
   2. Select Connections in the navigation menu.
   3. Enable Direct Connections by turning on the toggle. This allows users to connect to their own OpenAI compatible API endpoints.
2. Connect Open WebUI to Foundry Local:
   1. Select Settings in the profile menu.
   2. Select Connections in the navigation menu.
   3. Select + by Manage Direct Connections.
   4. For the URL, enter http://localhost:PORT/v1 where PORT is the Foundry Local endpoint port (use the CLI command foundry service status to find it). Note that Foundry Local dynamically assigns a port, so it isn't always the same.
   5. For the Auth, select None.
   6. Select Save

➡️ What is Foundry Local https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/what-is-foundry-local

➡️ Open WebUI: https://docs.openwebui.com/