When enterprises move generative AI from demos to real business workloads, the hardest question is usually not whether a model can answer a prompt. The harder question is whether the whole system can run reliably, predictably, securely, and economically over time. This becomes especially important as major model providers continue to adjust token pricing, context-window pricing, batching discounts, and model tiering.
That is where AI Runway becomes valuable. It turns model deployment into a Kubernetes-native platform capability. Instead of binding every application to a specific inference runtime, AI Runway lets teams describe model-serving intent through a unified ModelDeployment resource, while the platform selects or delegates to the right provider and engine underneath.
For teams already using Kubernetes, AKS, or cloud-native platform engineering practices, AI Runway offers a practical path from “calling an external model API” to “operating an enterprise inference platform.”
Why do we need a self-hosted inference platform?
Many teams have already proven the value of LLMs in knowledge assistants, code generation, content creation, customer support, document processing, and agentic workflows. But once usage grows, several platform-level issues appear quickly.
1. Token cost becomes an engineering problem
In a proof of concept, token usage often looks like a small budget line. In production, it becomes an architectural concern. A single RAG request may include system prompts, user input, retrieved context, tool outputs, and the final answer. An agentic workflow may call models many times for planning, routing, summarization, validation, and generation. An internal Copilot used by hundreds of employees can generate token consumption at a scale that surprises the original project team.
External model API cost is also affected by model versions, input/output token ratios, context length, caching policies, batch processing, and provider pricing strategy. When model vendors change pricing, enterprises without an alternative path become price takers.
Self-hosted inference does not mean replacing every external model. It means creating a controllable platform layer for high-frequency, predictable, localized, or privacy-sensitive workloads.
| Scenario | Why self-hosted inference helps |
|---|---|
| High-frequency internal Q&A | Large request volume can be served by smaller or quantized models |
| Document summarization and extraction | Stable task pattern, suitable for specialized local models |
| Agent intermediate steps | Planning, classification, and rewriting may not require the strongest closed model |
| Edge or private-network workloads | Data may need to stay inside a controlled boundary |
| Cost-sensitive applications | CPU/GPU resource pools, batching, and model tiering can reduce unit cost |
2. Data boundaries and compliance become clearer
Many enterprises are willing to use cloud-hosted models, but they also need clear controls for data classification, access boundaries, logging, and auditing. A self-hosted inference platform allows sensitive documents, internal knowledge bases, customer interactions, and business context to remain inside a governed network and operational model.
3. Teams should not be locked into one engine
Inference engines are evolving quickly. vLLM, SGLang, TensorRT-LLM, and llama.cpp serve different needs. Some are optimized for high-throughput GPU serving. Some are better for structured serving or NVIDIA GPU acceleration. Others make GGUF quantized models practical on CPU or lightweight GPU environments. A platform should not force every team into one runtime. It should provide a unified entry point and absorb runtime differences underneath.
4. Production AI requires model operations, not just endpoints
Production workloads need deployment lifecycle management, status, logs, metrics, scaling, debugging, progressive rollout, resource quotas, and secure ingress. A self-hosted inference platform should prevent every team from handcrafting runtime-specific YAML and instead provide these capabilities as shared platform primitives.
What is AI Runway?
AI Runway is a Kubernetes-native platform for deploying and managing large language models. Its core idea is to describe model deployment intent through a unified Kubernetes CRD called ModelDeployment. The AI Runway Controller then selects or delegates to provider-specific controllers based on provider capabilities.
The project describes itself as:
Deploy and manage large language models on Kubernetes — no YAML required.
AI Runway supports a Web UI, REST API, Headlamp Plugin, and direct CRD management with kubectl. The UI is optional and replaceable; the core platform capability lives in the controller, CRDs, and provider abstraction.
Key capabilities
| Capability | Value |
|---|---|
| Unified ModelDeployment CRD | One API for model, engine, resources, scaling, and gateway configuration |
| Multiple providers | Supports KAITO, NVIDIA Dynamo, KubeRay, llm-d, and provider shims |
| Multiple engines | Supports vLLM, SGLang, TensorRT-LLM, and llama.cpp |
| Automatic provider and engine selection | Matches CPU/GPU requirements, serving mode, and provider capability |
| Web UI and Headlamp Plugin | Simplifies model discovery, deployment, and monitoring |
| Hugging Face integration | Enables model catalog browsing and deployment |
| Observability | Surfaces deployment status, logs, and Prometheus metrics |
| Gateway API integration | Enables unified OpenAI-compatible routing through a gateway |
| Cost and capacity guidance | Helps with GPU fit, pricing, and capacity decisions |
Multi-engine support is the key differentiator
AI Runway is not just another model deployment tool. Its most important value is decoupling application developers from inference runtime decisions. Applications can call an OpenAI-compatible endpoint or a unified gateway, while the platform decides which engine and provider should serve a particular model.
| Engine | Typical use case | Resource target |
|---|---|---|
| vLLM | High-throughput general LLM serving | GPU |
| SGLang | Complex inference workflows and structured serving | GPU |
| TensorRT-LLM | Highly optimized inference on NVIDIA GPUs | GPU |
| llama.cpp | GGUF quantized models and lightweight inference | CPU / GPU |
For teams, this is an important story: instead of forcing every team into the same runtime, AI Runway creates a common platform where different workloads can choose different engines while keeping the developer experience consistent.
AI Runway architecture overview
The following Mermaid diagram shows a simplified view of the AI Runway platform layers.
Three design points matter most:
- Unified control plane: users submit ModelDeployment resources instead of handcrafting YAML for each runtime.
- Out-of-tree providers: KAITO, Dynamo, KubeRay, and llm-d declare their capabilities through provider shims and controllers.
- Replaceable runtime layer: the same platform can serve CPU-based llama.cpp models and GPU-based vLLM or TensorRT-LLM workloads.
Solution 1: Local Kubernetes with AI Runway, KAITO, and CPU
Local Kubernetes is ideal for learning, demos, development validation, and small-model prototyping. The goal is not maximum throughput. The goal is to prove that AI Runway + KAITO + llama.cpp can expose an OpenAI-compatible model service without requiring a GPU.
When to use this pattern
| Scenario | Description |
|---|---|
| Local developer experiments | Use kind, minikube, k3d, or Docker Desktop Kubernetes |
| Platform demos | Show the ModelDeployment, provider, and OpenAI-compatible API flow |
| CPU-only validation | No GPU or cloud resource required |
| SLM / GGUF testing | Use llama.cpp to serve quantized models |
For local CPU inference, allocate at least 4 vCPU and 12 GiB memory. Even small models need memory for runtime startup, model loading, KV cache, and context windows.
Local architecture
The local KAITO + CPU pattern is powerful for education and adoption:
- Developers learn the ModelDeployment abstraction without needing a GPU.
- The application does not need to know whether the backend is LocalAI, llama.cpp, or KAITO Workspace.
- CPU-only environments can still run lightweight and quantized models.
- Teams can validate models, prompts, and API behavior locally before moving to AKS or production clusters.
Sample Guideline - https://gist.github.com/kinfey/28b2338845cc63139aee2ea462a3c466
Solution 2: Azure with AKS, AI Runway, KAITO, and CPU
After local validation, the next step is usually a cloud-hosted inference platform. AKS provides managed Kubernetes control plane, node pools, networking, identity, monitoring, and Azure ecosystem integration. It is a natural foundation for AI Runway in production or pre-production environments.
The example below uses CPU-only AKS + KAITO + Qwen3-0.6B GGUF to build a cloud-hosted inference service without GPU nodes.
Azure architecture
Production recommendations for AKS
| Area | Recommendation |
|---|---|
| Secure ingress | Do not expose plain HTTP 80 directly; add TLS, API keys, OAuth2 Proxy, WAF, or internal LoadBalancer |
| Model governance | Pin model versions, image versions, and GGUF filenames |
| Cost governance | Use CPU for lightweight tasks and GPU for high-throughput large models |
| Observability | Integrate Azure Monitor, Prometheus, logs, and request-level metrics |
| Quota planning | Check regional vCPU/GPU quota before deployment |
| Caching | Use PVCs or model cache volumes to reduce repeated downloads |
| GitOps | Manage ModelDeployment, providers, and ingress through GitOps |
| Access control | Use namespaces, RBAC, and NetworkPolicy for team isolation |
Sample Guideline - https://gist.github.com/kinfey/d439a545d8c93e15d8a2854b65f03d4d
How to evangelize AI Runway inside an engineering organization
When introducing AI Runway, I would avoid starting with “we are building our own model platform.” A more effective narrative is:
- Start with cost predictability: high-frequency workloads should not all depend on the most expensive external model tier.
- Emphasize technical optionality: teams can use different models and engines while keeping a unified platform entry point.
- Highlight Kubernetes-native operations: existing AKS, RBAC, monitoring, GitOps, networking, and security practices can be reused.
- Use CPU demos to lower the barrier: local KAITO + CPU lets developers understand the full flow without GPUs.
- Use Azure as the production landing zone: AKS carries the same abstraction into cloud environments and can evolve toward GPU, gateway, monitoring, and multi-tenant governance.
This path avoids starting with GPU procurement, complex scheduling, or full-scale platform governance. Start small, prove the abstraction, then add higher-performance engines and stronger governance as the platform matures.
Closing thoughts
As AI applications enter production, enterprises need more than a model that can answer prompts. They need an inference platform that is controllable, observable, scalable, and evolvable. AI Runway brings this problem back into the Kubernetes platform engineering world: use ModelDeployment to standardize model deployment, use providers to hide runtime differences, and use multiple engines to match different cost and performance goals.
From a local Kubernetes KAITO + CPU demo to a Qwen3-0.6B CPU inference service on AKS, AI Runway provides a clear adoption path: start with a low-barrier setup, then evolve toward multi-model, multi-engine, multi-provider, unified-gateway, enterprise-governed inference.
In a world where token pricing changes frequently and model ecosystems evolve rapidly, a self-hosted inference platform is not about rejecting external models. It is about giving engineering teams more control over cost, architecture, and technical choice.
References
- AI Runway GitHub: https://github.com/kaito-project/airunway
- AI Runway Architecture: https://github.com/kaito-project/airunway/blob/main/docs/architecture.md
- AI Runway Providers: https://github.com/kaito-project/airunway/blob/main/docs/providers.md
- AI Runway CRD Reference: https://github.com/kaito-project/airunway/blob/main/docs/crd-reference.md
- KAITO: https://github.com/kaito-project/kaito
- LocalAI: https://localai.io
- AKS Application Routing: https://learn.microsoft.com/azure/aks/app-routing
- Qwen3-0.6B GGUF: https://huggingface.co/Qwen/Qwen3-0.6B-GGUF