Introduction

Deploying an AI application to Azure Container Apps is fundamentally different from deploying a web API. When you containerize a Django REST API, the application starts in a few seconds, the memory footprint is predictable, and the CPU usage scales linearly with requests. When you containerize a PyTorch model server, a LangChain agent, or an ONNX inference service, you are dealing with a completely different category of problem.

 

Large language models, computer vision models, and embedding pipelines can take minutes to load, consume gigabytes of memory before serving a single request, and produce bizarre errors when they encounter resource limits that look nothing like a standard out-of-memory exception. Add to that the challenge of running GPU workloads (or simulating them on CPU) in a containerized environment, and you have a troubleshooting landscape that catches even experienced ML engineers off guard.

 

This part of the series covers the real-world scenarios you will encounter when running AI workloads on Azure Container Apps, with specific focus on what goes wrong during deployment and startup — and how to fix it methodically.

 

Scenario 1: The Model Takes 3+ Minutes to Load and the Container Gets Killed Before It Starts

 What You See

You deploy your model inference service. In the logs you can see it is downloading or loading the model from disk:

 

INFO: Loading model from /app/models/model.bin

INFO: Loading tokenizer...

INFO: Loading weights layer 0/48...

INFO: Loading weights layer 12/48...

 

And then, abruptly, the container disappears and a new one starts. The liveness or startup probe has timed out and Container Apps has killed the container before the model finished loading. You end up in an endless restart loop where the model never fully loads.

Why This Happens

The default probe configuration does not account for long model loading times. The liveness probe begins firing almost immediately after the container starts. If your model takes 3 minutes to load and your liveness probe allows only 30 seconds of failure before killing the container, the model never gets a chance to finish loading. Container Apps is doing exactly what it was configured to do — it just was not configured with your workload in mind.

There is a second, related problem: if your model file is downloaded at container startup (from Azure Blob Storage, Hugging Face Hub, or a mounted file share), the download time is added on top of the load time, making the window even wider.

Step-by-Step Fix

Step 1 — Separate your startup probe from your liveness probe.

A startup probe fires repeatedly until it succeeds, and while it is in progress, the liveness probe is suppressed. This gives your model the time it needs to load without the risk of being killed. Set a generous `failureThreshold` and `periodSeconds`:

probes:
  - type: Startup
    httpGet:
      path: /health/startup
      port: 8080
    initialDelaySeconds: 10
    periodSeconds: 15
    failureThreshold: 40    # 40 * 15s = 600 seconds (10 minutes) for model to load
  - type: Liveness
    httpGet:
      path: /health/live
      port: 8080
    periodSeconds: 30
    failureThreshold: 3
  - type: Readiness
    httpGet:
      path: /health/ready
      port: 8080
    periodSeconds: 10
    failureThreshold: 3

 

Step 2 — Implement a proper three-tier health endpoint in your model server.

Each probe endpoint should return the appropriate status based on what it knows about the model:

# FastAPI model server with staged health endpoints
from fastapi import FastAPI, HTTPException
from contextlib import asynccontextmanager
import asyncio

model = None
model_loaded = False

@asynccontextmanager
async def lifespan(app: FastAPI):
    global model, model_loaded
    # Model loads in the background so the HTTP server starts immediately
    asyncio.create_task(load_model_async())
    yield
    # Cleanup on shutdown
    model = None

app = FastAPI(lifespan=lifespan)

async def load_model_async():
    global model, model_loaded
    import logging
    logger = logging.getLogger(__name__)   

    logger.info("Starting model load...")
    try:
        # Import heavy libraries only when needed
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer
     
        tokenizer = AutoTokenizer.from_pretrained("/app/models/my-model")
        model_obj = AutoModelForCausalLM.from_pretrained(
            "/app/models/my-model",
            torch_dtype=torch.float16,
            device_map="auto"
        )

        model = {"tokenizer": tokenizer, "model": model_obj}
        model_loaded = True
        logger.info("Model loaded successfully")
    except Exception as e:
        logger.error(f"Model loading failed: {e}", exc_info=True)
        raise

@app.get("/health/startup")
def startup_probe():
    # Returns 200 immediately — the container is alive but model may still be loading
    return {"status": "starting"}

@app.get("/health/live")
def liveness_probe():
    # Returns 200 as long as the process has not entered a broken state
    return {"status": "alive"}

@app.get("/health/ready")

def readiness_probe():
    # Returns 200 ONLY when the model is fully loaded
    if not model_loaded:
        raise HTTPException(status_code=503, detail="Model not yet loaded")
    return {"status": "ready"}

 

Step 3 — Pre-download model weights into the container image at build time.

Downloading models at container startup is a significant reliability and performance risk. If the Hugging Face Hub or your storage account is temporarily unreachable, the container cannot start. Instead, bake the model weights directly into the image or use a separate initialization Container App Job to pre-populate a persistent volume:

# Option A: Bake the model into the image (simple, but creates a large image)
FROM python:3.11-slim
WORKDIR /app
RUN pip install transformers torch --index-url https://download.pytorch.org/whl/cpu
# Download model at build time
RUN python -c "from transformers import AutoTokenizer, AutoModel; 
    AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models'); 
    AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models')"
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

# Option B: Pre-populate an Azure Files volume using a Container App Job
az containerapp job create 
  --name model-downloader-job 
  --resource-group my-rg 
  --environment my-aca-env 
  --trigger-type Manual 
  --replica-timeout 600 
  --image python:3.11-slim 
  --command "bash" 
  --args "-c" "pip install huggingface_hub && huggingface-cli download my-org/my-model --local-dir /mnt/models" 
  --volume-mounts "model-storage:/mnt/models" 
  --volumes "model-storage:azureFile:my-fileshare"

 

 Scenario 2: The Model Server Runs Out of Memory and Gets OOM-Killed

What You See

Your model server starts successfully in development with 8 GB of RAM. In Azure Container Apps with a 4.0 vCPU / 8.0 Gi configuration (the maximum without GPU support), it crashes intermittently. The container restarts with no error in the application logs. When you check the system logs, you see the container exit code is `137`.

Exit code 137 indicates the process was killed by the kernel OOM killer.

Or in Log Analytics:

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "my-model-server"
| where Log_s contains "OOMKilled" or Log_s contains "137"
| project TimeGenerated, Log_s

Why This Happens

 

Exit code 137 means the container was sent SIGKILL (`128 + 9 = 137`) because it exceeded its memory limit. Container Apps enforces memory limits strictly. When the process tries to allocate more memory than the container is allowed, the Linux kernel's OOM (Out Of Memory) killer terminates the process.

With ML models, memory usage is not constant. A model might use 4 GB at rest but spike to 7 GB during inference when the input batch is large, when attention maps are being computed, or when the model is warming up its KV cache. If your container limit is 8 Gi and the model uses 7 Gi at rest plus 2 Gi during inference, you will hit OOM under load even if the container "usually" has enough memory.

Step-by-Step Fix

Step 1 — Quantize your model to reduce its memory footprint.

Full-precision (FP32) models use twice the memory of half-precision (FP16) models, and INT8 quantized models use half the memory of FP16. For inference workloads where a small accuracy trade-off is acceptable, quantization is the single most impactful optimization you can make:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# 4-bit quantization — reduces a 7B parameter model from ~14GB to ~4GB
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "my-model-path",
    quantization_config=quantization_config,
    device_map="auto"
)

 

Step 2 — Implement request batching to flatten memory spikes.

 

Processing one request at a time with a large model creates memory spikes every time a new request arrives. Batching requests together means the model stays in a steady memory state rather than spiking per request:

from asyncio import Queue, wait_for
import asyncio

class BatchedInferenceService:
    def __init__(self, model, tokenizer, max_batch_size=4, timeout=0.05):
        self.model = model
        self.tokenizer = tokenizer
        self.max_batch_size = max_batch_size
        self.timeout = timeout
        self.queue = Queue()
     

    async def infer(self, text: str) -> str:
        future = asyncio.get_event_loop().create_future()
        await self.queue.put((text, future))
        return await future
  
    async def process_batches(self):
        while True:
            batch = []
            futures = []
          
            # Wait for the first item
            text, future = await self.queue.get()
            batch.append(text)
            futures.append(future)
          
            # Try to collect more items within the timeout window
            try:
                while len(batch) < self.max_batch_size:
                    text, future = await wait_for(self.queue.get(), timeout=self.timeout)
                    batch.append(text)
                    futures.append(future)
            except asyncio.TimeoutError:
                pass
          
            # Process the whole batch at once
            try:
                results = self._run_inference_batch(batch)
                for future, result in zip(futures, results):
                    future.set_result(result)
            except Exception as e:
                for future in futures:
                    future.set_exception(e)   

    def _run_inference_batch(self, texts):
        with torch.no_grad():
            inputs = self.tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
            outputs = self.model(**inputs)
            return outputs.logits.tolist()

 

Step 3 — Set memory limits explicitly in your Container App to match what you expect at peak.

 

Do not let the container use all available node memory and get killed unexpectedly. Set explicit limits that match your model's peak usage:

# For a model that uses 5.5 GB at peak, allocate 6 GB
az containerapp update 
  --name my-model-server 
  --resource-group my-rg 
  --cpu 2.0 
  --memory 4.0Gi

 

Step 4 — Add memory monitoring to your health endpoint.

 

Your readiness probe should check available memory and return 503 if memory pressure is too high, which will cause the load balancer to route traffic to other healthy replicas:

import psutil

@app.get("/health/ready")
def readiness_probe():
    if not model_loaded:
        raise HTTPException(status_code=503, detail="Model not yet loaded")
   
    memory = psutil.virtual_memory()
    if memory.percent > 90:
        raise HTTPException(
            status_code=503,
            detail=f"Memory pressure too high: {memory.percent:.1f}% used"
        )
  
    return {
        "status": "ready",
        "memory_percent": memory.percent,
        "available_gb": memory.available / (1024**3)
    }

 

Scenario 3: GPU Workloads Fail to Initialize

What You See

 

You deploy a container that uses PyTorch with CUDA support. The container starts, but you see errors like:

CUDA error: no kernel image is available for execution on the device

torch.cuda.is_available() returned False

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB

 

Or the model silently falls back to CPU without telling you, and your inference times are 50x slower than expected.

Why This Happens

Azure Container Apps supports GPU workloads through a dedicated GPU consumption plan and specialized GPU-enabled environments. If you deploy a CUDA-enabled container to a standard Container Apps environment (which uses CPU-only nodes), `torch.cuda.is_available()` returns `False` and PyTorch either errors out or silently falls back to CPU depending on how your code handles it.

Even when you are in a GPU-enabled environment, CUDA version mismatches between the CUDA toolkit installed in your container image and the CUDA drivers on the host node will produce the "no kernel image" error.

Step-by-Step Fix

Step 1 — Detect whether GPU is available and log it explicitly at startup.

Never assume — always log the device your model is using:

import torch
import logging
logger = logging.getLogger(__name__)

def initialize_device():
    if torch.cuda.is_available():
        device = torch.device("cuda")
        logger.info(f"Using GPU: {torch.cuda.get_device_name(0)}")
        logger.info(f"CUDA version: {torch.version.cuda}")
        logger.info(f"Available GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
    else:
        device = torch.device("cpu")
        logger.warning("GPU not available, falling back to CPU. Inference will be slower.")
    return device
device = initialize_device()
model = model.to(device)

 

Step 2 — Match your CUDA toolkit version to the host driver.

The CUDA toolkit version in your image must be less than or equal to the CUDA driver version on the host. Use the official PyTorch images which bundle compatible CUDA versions:

# Use PyTorch's official image with a specific CUDA version
# Check Azure Container Apps GPU documentation for supported CUDA versions
FROM pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

 

Step 3 — Create your Container Apps environment with GPU support enabled.

 

GPU support requires a dedicated GPU workload profile in your Container Apps environment:

# Create an environment with GPU workload profil

az containerapp env create 
  --name my-gpu-env 
  --resource-group my-rg 
  --location eastus 
  --workload-profile-type "NC24-A100" 
  --workload-profile-name "gpu-profile"

# Deploy your app to the GPU profile
az containerapp create 
  --name my-model-server 
  --resource-group my-rg 
  --environment my-gpu-env 
  --image myregistry.azurecr.io/my-model-server:cuda11.8 
  --cpu 4.0 
  --memory 16.0Gi 
  --workload-profile-name "gpu-profile" 
  --min-replicas 1

 

Step 4 — Handle the CPU fallback gracefully so you know when it happens.

 

If you want your application to work on both GPU and CPU environments, implement a clean fallback that makes the degraded state visible in monitoring:

import os
import torch

class ModelConfig:
    def __init__(self):
        self.force_cpu = os.environ.get("FORCE_CPU", "false").lower() == "true"
        self.device = self._select_device()
       
    def _select_device(self):
        if self.force_cpu:
            return torch.device("cpu")
       
        if torch.cuda.is_available():
            return torch.device("cuda")
      
        # GPU was expected but not found — emit a warning metric
        import warnings
        warnings.warn(
            "CUDA requested but not available. Running on CPU. "
            "Inference latency will be significantly higher.",
            RuntimeWarning
        )
        # Optionally: push a custom metric to Azure Monitor here
        return torch.device("cpu")
   
    @property
    def is_gpu(self):
        return self.device.type == "cuda"

 Scenario 4: LangChain / AI Agent Timeouts at Startup

What You See

You deploy a LangChain-based agent or a RAG (Retrieval-Augmented Generation) pipeline to Container Apps. During startup, the application connects to Azure OpenAI, loads an embedding model, and populates an in-memory vector store. But the readiness probe times out while the vector store is being populated from a large document set.

Why This Happens

LangChain applications often do expensive work at startup — embedding thousands of documents, pre-populating vector indexes, or loading conversation history from a database. This work happens synchronously in many LangChain components, blocking the main thread and preventing the HTTP server from responding to health probes.

Step-by-Step Fix

Step 1 — Move initialization work to a background task that runs after the HTTP server is up

 

from fastapi import FastAPI
from contextlib import asynccontextmanager
import asyncio
import logging

logger = logging.getLogger(__name__)
vector_store = None
initialization_complete = False
initialization_error = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global initialization_complete, initialization_error   

    # Start the heavy initialization in the background
    task = asyncio.create_task(initialize_vector_store())   

    yield  # Server starts and begins serving health probe requests  

    # Cleanup
    task.cancel()

app = FastAPI(lifespan=lifespan)

async def initialize_vector_store():
    global vector_store, initialization_complete, initialization_error
    try:
        logger.info("Starting vector store initialization...")      

        # Run CPU-bound work in a thread pool so we don't block the event loop
        loop = asyncio.get_event_loop()
        vector_store = await loop.run_in_executor(None, _build_vector_store)       

        initialization_complete = True
        logger.info("Vector store initialization complete")
    except Exception as e:
        initialization_error = str(e)
        logger.error(f"Vector store initialization failed: {e}", exc_info=True)

def _build_vector_store():
    from langchain_community.vectorstores import FAISS
    from langchain_openai import AzureOpenAIEmbeddings
    from langchain_community.document_loaders import DirectoryLoader  

    loader = DirectoryLoader("/app/documents")
    documents = loader.load()   

    embeddings = AzureOpenAIEmbeddings(
        azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"],
        azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        api_key=os.environ["AZURE_OPENAI_API_KEY"]
    )
   
    return FAISS.from_documents(documents, embeddings)

@app.get("/health/ready")
def readiness():
    if initialization_error:
        raise HTTPException(status_code=500, detail=f"Initialization failed: {initialization_error}")
    if not initialization_complete:
        raise HTTPException(status_code=503, detail="Initializing vector store, please wait...")
    return {"status": "ready"}

 

Step 2 — Use Azure AI Search instead of an in-memory vector store for large document sets.

 

In-memory vector stores like FAISS are fine for development but become a liability in production Container Apps because the index is lost every time the container restarts, and rebuilding it adds minutes to your startup time. Azure AI Search persists the index and provides near-instant load times:

from langchain_community.vectorstores.azuresearch import AzureSearch
from langchain_openai import AzureOpenAIEmbeddings

embeddings = AzureOpenAIEmbeddings(
    azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"]
)

# The index already exists in Azure AI Search — no rebuild needed on startup
vector_store = AzureSearch(
    azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
    azure_search_key=os.environ["AZURE_SEARCH_KEY"],
    index_name="my-document-index",
    embedding_function=embeddings.embed_query
)

# This is nearly instantaneous — just connects to the existing index
initialization_complete = True

Debugging AI Workload Logs in Log Analytics

 

When something goes wrong with your AI workload, these Log Analytics queries will help you quickly identify the pattern:

 

// Find all OOM kills in the last 24 hours
ContainerAppSystemLogs_CL
| where TimeGenerated > ago(24h)
| where Log_s contains "OOMKilled" or ExitCode_d == 137
| project TimeGenerated, ContainerAppName_s, Log_s, ExitCode_d
| order by TimeGenerated desc

// Track model loading time across restarts
ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "my-model-server"
| where Log_s contains "Model loaded" or Log_s contains "Starting model load"
| project TimeGenerated, Log_s, ContainerName_s
| order by TimeGenerated asc

// Find slow inference requests (if you are logging inference latency)
ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "my-model-server"
| where Log_s contains "inference_latency_ms"
| extend latency = toint(extract("inference_latency_ms=([0-9]+)", 1, Log_s))
| where latency > 5000    // Requests taking more than 5 seconds
| project TimeGenerated, latency, Log_s
| order by latency desc

 

 Summary: AI Workload Startup Checklist

 

When your AI workload fails to start or behaves unexpectedly, work through this checklist:

 

- Is the model loading time covered by an appropriate startup probe `failureThreshold`?

- Are model weights baked into the image or pre-loaded onto a mounted volume, rather than downloaded at runtime?

- Is the container's memory limit large enough for peak inference load (model size + activation memory)?

- Have you verified whether the workload is running on GPU or CPU and logged that explicitly at startup?

- Does the CUDA version in your image match or precede the driver version on the host?

- For LangChain or RAG workloads, does initialization happen in the background so health probes can respond?

- Are you using a persistent vector store (Azure AI Search, Azure Cosmos DB) instead of rebuilding in-memory indexes on every restart?

References and Sample Resources

Use these links alongside the scenarios above to go deeper on configuration details and production patterns.

Azure Container Apps docs (core)

• Health probes in Azure Container Apps
• Workload profiles overview (Consumption, Dedicated, Memory-optimized, GPU)
• Serverless GPU overview and supported regions
• Azure Container Apps Jobs (for preloading models or maintenance tasks)
• Monitoring and logging options in Azure Container Apps

AI-specific docs

• AI integration with Azure Container Apps
• LangChain tutorial with Container Apps dynamic sessions
• Azure AI Search vector search overview
• Azure OpenAI with Python quickstart

Official sample repositories

• Chat app with Azure OpenAI on Container Apps
• AI MCP server sample on Container Apps
• .NET MCP agent app (client/server pattern on Container Apps)
• TypeScript MCP container sample
• Python code interpreter dynamic session sample

ML runtime and model-loading references

• PyTorch CUDA notes and best practices
• Hugging Face Transformers loading models from local/Hub paths
• bitsandbytes quantization guide

What's Next

 

In the final part of this series, we step back from reactive troubleshooting and look at how to build a proactive observability and automation layer so that you catch these problems before your users do — or better yet, have them resolve themselves automatically.

 

 

Part of the series: Troubleshooting Azure Container Apps in Production

Next: Part 4 — Eyes Open, Hands Free: Automating Observability, Alerts, and Self-Healing Diagnostics for Azure Container Apps