Blog Post

Microsoft Developer Community Blog
7 MIN READ

Building a Controllable Inference Platform on Kubernetes with AI Runway

kinfey's avatar
kinfey
Icon for Microsoft rankMicrosoft
May 18, 2026

When enterprises move generative AI from demos to real business workloads, the hardest question is usually not whether a model can answer a prompt. The harder question is whether the whole system can run reliably, predictably, securely, and economically over time. This becomes especially important as major model providers continue to adjust token pricing, context-window pricing, batching discounts, and model tiering.

That is where AI Runway becomes valuable. It turns model deployment into a Kubernetes-native platform capability. Instead of binding every application to a specific inference runtime, AI Runway lets teams describe model-serving intent through a unified ModelDeployment resource, while the platform selects or delegates to the right provider and engine underneath.

For teams already using Kubernetes, AKS, or cloud-native platform engineering practices, AI Runway offers a practical path from “calling an external model API” to “operating an enterprise inference platform.”

Why do we need a self-hosted inference platform?

Many teams have already proven the value of LLMs in knowledge assistants, code generation, content creation, customer support, document processing, and agentic workflows. But once usage grows, several platform-level issues appear quickly.

1. Token cost becomes an engineering problem

In a proof of concept, token usage often looks like a small budget line. In production, it becomes an architectural concern. A single RAG request may include system prompts, user input, retrieved context, tool outputs, and the final answer. An agentic workflow may call models many times for planning, routing, summarization, validation, and generation. An internal Copilot used by hundreds of employees can generate token consumption at a scale that surprises the original project team.

External model API cost is also affected by model versions, input/output token ratios, context length, caching policies, batch processing, and provider pricing strategy. When model vendors change pricing, enterprises without an alternative path become price takers.

Self-hosted inference does not mean replacing every external model. It means creating a controllable platform layer for high-frequency, predictable, localized, or privacy-sensitive workloads.

ScenarioWhy self-hosted inference helps
High-frequency internal Q&ALarge request volume can be served by smaller or quantized models
Document summarization and extractionStable task pattern, suitable for specialized local models
Agent intermediate stepsPlanning, classification, and rewriting may not require the strongest closed model
Edge or private-network workloadsData may need to stay inside a controlled boundary
Cost-sensitive applicationsCPU/GPU resource pools, batching, and model tiering can reduce unit cost

2. Data boundaries and compliance become clearer

Many enterprises are willing to use cloud-hosted models, but they also need clear controls for data classification, access boundaries, logging, and auditing. A self-hosted inference platform allows sensitive documents, internal knowledge bases, customer interactions, and business context to remain inside a governed network and operational model.

3. Teams should not be locked into one engine

Inference engines are evolving quickly. vLLM, SGLang, TensorRT-LLM, and llama.cpp serve different needs. Some are optimized for high-throughput GPU serving. Some are better for structured serving or NVIDIA GPU acceleration. Others make GGUF quantized models practical on CPU or lightweight GPU environments. A platform should not force every team into one runtime. It should provide a unified entry point and absorb runtime differences underneath.

4. Production AI requires model operations, not just endpoints

Production workloads need deployment lifecycle management, status, logs, metrics, scaling, debugging, progressive rollout, resource quotas, and secure ingress. A self-hosted inference platform should prevent every team from handcrafting runtime-specific YAML and instead provide these capabilities as shared platform primitives.

What is AI Runway?

AI Runway is a Kubernetes-native platform for deploying and managing large language models. Its core idea is to describe model deployment intent through a unified Kubernetes CRD called ModelDeployment. The AI Runway Controller then selects or delegates to provider-specific controllers based on provider capabilities.

The project describes itself as:

Deploy and manage large language models on Kubernetes — no YAML required.

AI Runway supports a Web UI, REST API, Headlamp Plugin, and direct CRD management with kubectl. The UI is optional and replaceable; the core platform capability lives in the controller, CRDs, and provider abstraction.

Key capabilities

CapabilityValue
Unified ModelDeployment CRDOne API for model, engine, resources, scaling, and gateway configuration
Multiple providersSupports KAITO, NVIDIA Dynamo, KubeRay, llm-d, and provider shims
Multiple enginesSupports vLLM, SGLang, TensorRT-LLM, and llama.cpp
Automatic provider and engine selectionMatches CPU/GPU requirements, serving mode, and provider capability
Web UI and Headlamp PluginSimplifies model discovery, deployment, and monitoring
Hugging Face integrationEnables model catalog browsing and deployment
ObservabilitySurfaces deployment status, logs, and Prometheus metrics
Gateway API integrationEnables unified OpenAI-compatible routing through a gateway
Cost and capacity guidanceHelps with GPU fit, pricing, and capacity decisions

Multi-engine support is the key differentiator

AI Runway is not just another model deployment tool. Its most important value is decoupling application developers from inference runtime decisions. Applications can call an OpenAI-compatible endpoint or a unified gateway, while the platform decides which engine and provider should serve a particular model.

EngineTypical use caseResource target
vLLMHigh-throughput general LLM servingGPU
SGLangComplex inference workflows and structured servingGPU
TensorRT-LLMHighly optimized inference on NVIDIA GPUsGPU
llama.cppGGUF quantized models and lightweight inferenceCPU / GPU

For teams, this is an important story: instead of forcing every team into the same runtime, AI Runway creates a common platform where different workloads can choose different engines while keeping the developer experience consistent.

AI Runway architecture overview

The following Mermaid diagram shows a simplified view of the AI Runway platform layers.

Three design points matter most:

  1. Unified control plane: users submit ModelDeployment resources instead of handcrafting YAML for each runtime.
  2. Out-of-tree providers: KAITO, Dynamo, KubeRay, and llm-d declare their capabilities through provider shims and controllers.
  3. Replaceable runtime layer: the same platform can serve CPU-based llama.cpp models and GPU-based vLLM or TensorRT-LLM workloads.

 

Solution 1: Local Kubernetes with AI Runway, KAITO, and CPU

Local Kubernetes is ideal for learning, demos, development validation, and small-model prototyping. The goal is not maximum throughput. The goal is to prove that AI Runway + KAITO + llama.cpp can expose an OpenAI-compatible model service without requiring a GPU.

When to use this pattern

ScenarioDescription
Local developer experimentsUse kind, minikube, k3d, or Docker Desktop Kubernetes
Platform demosShow the ModelDeployment, provider, and OpenAI-compatible API flow
CPU-only validationNo GPU or cloud resource required
SLM / GGUF testingUse llama.cpp to serve quantized models

For local CPU inference, allocate at least 4 vCPU and 12 GiB memory. Even small models need memory for runtime startup, model loading, KV cache, and context windows.

Local architecture

The local KAITO + CPU pattern is powerful for education and adoption:

  • Developers learn the ModelDeployment abstraction without needing a GPU.
  • The application does not need to know whether the backend is LocalAI, llama.cpp, or KAITO Workspace.
  • CPU-only environments can still run lightweight and quantized models.
  • Teams can validate models, prompts, and API behavior locally before moving to AKS or production clusters.

Sample Guideline - https://gist.github.com/kinfey/28b2338845cc63139aee2ea462a3c466

Solution 2: Azure with AKS, AI Runway, KAITO, and CPU

After local validation, the next step is usually a cloud-hosted inference platform. AKS provides managed Kubernetes control plane, node pools, networking, identity, monitoring, and Azure ecosystem integration. It is a natural foundation for AI Runway in production or pre-production environments.

The example below uses CPU-only AKS + KAITO + Qwen3-0.6B GGUF to build a cloud-hosted inference service without GPU nodes.

Azure architecture

Production recommendations for AKS

AreaRecommendation
Secure ingressDo not expose plain HTTP 80 directly; add TLS, API keys, OAuth2 Proxy, WAF, or internal LoadBalancer
Model governancePin model versions, image versions, and GGUF filenames
Cost governanceUse CPU for lightweight tasks and GPU for high-throughput large models
ObservabilityIntegrate Azure Monitor, Prometheus, logs, and request-level metrics
Quota planningCheck regional vCPU/GPU quota before deployment
CachingUse PVCs or model cache volumes to reduce repeated downloads
GitOpsManage ModelDeployment, providers, and ingress through GitOps
Access controlUse namespaces, RBAC, and NetworkPolicy for team isolation


Sample Guideline - https://gist.github.com/kinfey/d439a545d8c93e15d8a2854b65f03d4d

How to evangelize AI Runway inside an engineering organization

When introducing AI Runway, I would avoid starting with “we are building our own model platform.” A more effective narrative is:

  1. Start with cost predictability: high-frequency workloads should not all depend on the most expensive external model tier.
  2. Emphasize technical optionality: teams can use different models and engines while keeping a unified platform entry point.
  3. Highlight Kubernetes-native operations: existing AKS, RBAC, monitoring, GitOps, networking, and security practices can be reused.
  4. Use CPU demos to lower the barrier: local KAITO + CPU lets developers understand the full flow without GPUs.
  5. Use Azure as the production landing zone: AKS carries the same abstraction into cloud environments and can evolve toward GPU, gateway, monitoring, and multi-tenant governance.

This path avoids starting with GPU procurement, complex scheduling, or full-scale platform governance. Start small, prove the abstraction, then add higher-performance engines and stronger governance as the platform matures.

Closing thoughts


As AI applications enter production, enterprises need more than a model that can answer prompts. They need an inference platform that is controllable, observable, scalable, and evolvable. AI Runway brings this problem back into the Kubernetes platform engineering world: use ModelDeployment to standardize model deployment, use providers to hide runtime differences, and use multiple engines to match different cost and performance goals.

From a local Kubernetes KAITO + CPU demo to a Qwen3-0.6B CPU inference service on AKS, AI Runway provides a clear adoption path: start with a low-barrier setup, then evolve toward multi-model, multi-engine, multi-provider, unified-gateway, enterprise-governed inference.

In a world where token pricing changes frequently and model ecosystems evolve rapidly, a self-hosted inference platform is not about rejecting external models. It is about giving engineering teams more control over cost, architecture, and technical choice.

References

Updated May 18, 2026
Version 1.0
No CommentsBe the first to comment