A Successful Demo Is Not a Production Platform
Getting GitHub Actions runners working on Azure Kubernetes Service (AKS) with Actions Runner Controller (ARC) is relatively straightforward. Within a short time, you can deploy ARC, register a runner scale set, and execute workflows successfully.
However, a successful runner only proves that the components can communicate.
It does not prove the platform is secure, supportable, observable, or ready to support multiple engineering teams.
In our experience, moving ARC from proof of concept to production was defined less by Kubernetes deployment and more by operational decisions around identity, network connectivity, workload isolation, runner images, and ownership.
🎯 Production Reality
Running a workflow successfully proves the deployment works.
Running hundreds of workflows across multiple engineering teams proves the platform works.
Identity Scope Is an Architecture Decision
Authentication should be treated as a platform design decision rather than a deployment parameter.
Repository- and organization-level runner scale sets typically use a GitHub App, while enterprise-level runner scale sets require a Personal Access Token (Classic).
Before deployment, establish:
- Where the runner scale set will be registered
- Which repositories can use the runner group
- How credentials will be stored and rotated
- Who owns authentication lifecycle management
The following values quickly become platform dependencies:
|
Placeholder |
Represents |
Example |
|
<GITHUB_CONFIG_URL> |
GitHub scope hosting runners |
https://github.com/contoso |
|
<RUNNER_GROUP> |
Repository access boundary |
aks-platform-runners |
|
<RUNNER_SCALE_SET_NAME> |
Workflow target name |
aks-arc-linux |
|
<CONTROLLER_NAMESPACE> |
ARC controller namespace |
arc-systems |
|
<RUNNER_NAMESPACE> |
Runner namespace |
arc-runners |
|
<RUNNER_IMAGE> |
Runner image reference |
contoso.azurecr.io/actions-runner:2.329.0 |
💡 Production Insight
The biggest challenge is rarely authentication itself.
It is defining ownership when credentials need to be rotated, audited, recovered, or replaced.
DNS and Egress Are Part of the Runner Platform
Many ARC deployments focus on Kubernetes configuration while underestimating network dependencies.
A workflow may authenticate successfully with GitHub and still fail because actions, artifacts, package feeds, registries, or deployment targets are unreachable.
In addition to workflow-specific dependencies, ensure that AKS outbound requirements for your chosen networking model are satisfied. AKS platform dependencies are separate from ARC and workflow connectivity requirements.
The table below provides a practical baseline for outbound connectivity validation.
|
Traffic Category |
Common Destinations |
Port |
Why It Matters |
|
GitHub API and Registration |
github.com, api.github.com |
TCP 443 |
Authentication and runner registration |
|
Actions Communication |
*.actions.githubusercontent.com |
TCP 443 |
Job execution and results |
|
Actions Downloads |
codeload.github.com, objects.githubusercontent.com |
TCP 443 |
Action and runner downloads |
|
GitHub Packages |
ghcr.io, *.pkg.github.com |
TCP 443 |
Images and packages |
|
Azure APIs |
management.azure.com, login.microsoftonline.com |
TCP 443 |
Azure deployments and authentication |
|
DNS Resolution |
Cluster DNS path |
UDP/TCP 53 |
Required for endpoint resolution |
|
Private Registry |
<CONTAINER_REGISTRY_FQDN> |
TCP 443 |
Runner image pulls |
|
Workflow Dependencies |
Package feeds, scanners, deployment targets |
Varies |
Workflow-specific requirements |
✅ Validation Tip
Always validate connectivity from the runner pod itself.
Validating from the AKS node does not guarantee workflow traffic can reach required endpoints.
Separate ARC Management from Workflow Execution
A production deployment should isolate platform components from workload execution.
A common architecture looks like:
The controller manages ARC resources, while ephemeral runner pods execute arbitrary workflow code.
For operational and security reasons, these workloads should not share the same boundary.
|
Component |
Namespace |
|
ARC Controller |
Dedicated management namespace |
|
Listeners and Runners |
Dedicated runner namespace |
Another important design choice is container execution strategy.
|
Model |
Benefits |
Considerations |
|
Docker-in-Docker |
Maximum workflow compatibility |
Privileged containers |
|
Kubernetes Mode |
Better isolation |
Additional complexity |
|
No Container Mode |
Simpler operations |
Limited workload support |
The right choice depends on workload requirements and security expectations.
Configuration Names Become Platform Contracts
One of the most overlooked production considerations is naming.
A runner scale set name is not merely a Helm value. It becomes an interface consumed by development teams through the runs-on configuration.
jobs:
plan:
name: ARC runner validation
runs-on: <RUNNER_SCALE_SET_NAME>
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Validate runner
run: |
echo "Running on an ARC self-hosted runner"
uname -a
The following quickly become platform contracts:
- Runner scale set names
- Runner groups
- Namespace boundaries
- Runner image references
Choose these names carefully and document which values are intended to remain stable.
A repository should not need to understand ARC internals. It should consume a predictable runner service.
Treat Runner Images as Managed Artifacts
Custom runner images often begin as a convenience and eventually become shared platform dependencies.
Treat them as managed platform artifacts.
Recommended practices include:
- Pin runner versions
- Pin base image versions
- Scan images before release
- Publish immutable tags or digests
- Validate representative workflows
- Maintain tested rollback versions
Once multiple repositories depend on a runner image, each image update effectively becomes a production release.
Design Observability Beyond Job Success
A successful workflow confirms execution, not operational readiness.
Monitor the entire platform, not only workflow outcomes.
At a minimum, retain logs for:
- ARC controller
- Listener components
- Runner pods
Monitor:
- Queue duration
- Pod provisioning latency
- Authentication failures
- Image pull failures
- Scheduling failures
Alerts should help distinguish between:
- GitHub-side issues
- AKS scheduling problems
- Capacity constraints
- DNS failures
- Identity failures
- Network connectivity issues
The objective is not simply knowing that a workflow failed but understanding exactly where it failed.
Production Readiness Checklist
Before onboarding additional teams, validate the following:
|
Area |
Validation |
|
Platform |
Controller healthy and runner pods created |
|
Scale Set |
Visible in expected GitHub scope |
|
Access |
Repository permissions validated |
|
Network |
GitHub, Azure, Registry and DNS reachable |
|
Images |
Runner image tested and scanned |
|
Execution |
Container execution model validated |
|
Observability |
Logs retained outside ephemeral runners |
|
Operations |
Rotation, rollback, ownership and escalation documented |
Troubleshoot ARC by Failure Boundary
When troubleshooting, start by identifying where workflow execution stopped.
|
Symptom |
Check |
Remediation |
|
Runners not visible in GitHub |
Authentication and runner-group access |
Validate credentials and scope |
|
Runner pods not created |
Scale set and controller status |
Inspect controller logs and events |
|
No matching runner available |
Scale set name and permissions |
Verify runner-group access |
|
Image pull failures |
Registry access and DNS |
Validate image path and connectivity |
|
Registration failures |
Authentication configuration |
Validate GitHub App or PAT settings |
From a Working Runner to a Dependable Platform
Our journey showed that deploying ARC is rarely the hardest part. The complexity emerges when runners become a shared service supporting multiple teams, environments, and business-critical workloads.
The organizations that succeed with ARC focus not only on deployment, but also on operational excellence, governance, reliability, and clear ownership from day one.
A green workflow run demonstrates capability. Consistent performance through outages, upgrades, scaling demands, and changing requirements demonstrates platform maturity.
"Deploying ARC is a project. Operating it is a platform responsibility."